Use GitLab platform token for container registry

[baurmatt]

What would you like Renovate to be able to do?

Using Renovate to update container images, that are hosted in private projects, currently requires the user to specify hostRules defining the credentials. At least on Gitlab, this might not be necessary if Renovate is already a member of the relevant project and the token has the right scope (read_repository).

We currently need to define the following hostRules for every repository:

  "hostRules": [
    {
      "description": "Personal token of the Gitlab renovate user with read_registry scope only.",
      "hostType": "docker",
      "matchHost": "registry.example.de",
      "username": "renovate-read-registry",
      "password": "glpat-xxxxxxxxxx"
    }
  ],

If you have any ideas on how this should be implemented, please tell us here.

Automatically use the Renovate token to authenticate against the container registry.

Is this a feature you are interested in implementing yourself?

No

[rarkins]

What's an example hostRules you need to add today manually?

[baurmatt]

@rarkins Thanks for your quick response! :) I've updated the description of this issue with the relevant hostrules.

[rarkins]

Questions:

1. If the registry is registry.example.de, what's the endpoint value you pass to Renovate?
2. Is the username a dummy string, or is that the real username which the PAT is created from?
3. Does this mean that the GitLab registry uses Basic Authorization instead of Bearer?

[baurmatt]

1. The endpoint value would be gitlab.example.de. I guess it would be good if the endpoint API could be used to retrieve the registry hostname? :)
2. In this case, it's the name of the PAT but it doesn't matter for Gitlab. It could be any random string.
3. I'm not 100% sure what Gitlab registry uses as an authentication method, but I'm pretty sure it's not basic auth :)

[baurmatt]

So best way I could find is this:

List registry repositories Within a project

-> https://docs.gitlab.com/ee/api/container_registry.html#within-a-project

$ curl -s --header "PRIVATE-TOKEN: $GITLAB_API_PRIVATE_TOKEN" "https://gitlab.example.de/api/v4/projects/mygroup%2Fterratest/registry/repositories" | jq '.'
[
  {
    "id": 470,
    "name": "",
    "path": "mygroup/terratest",
    "project_id": 5762,
    "location": "registry.example.de/mygroup/terratest",
    "created_at": "2021-11-05T13:56:11.890Z",
    "cleanup_policy_started_at": null
  }
]

Basically: Use the project full path to get the container repository of that project. The hostname of .[].location can be IMHO securely treated as the relevant registry.

[rarkins]

I think we might want to treat this as opt-in as it would involve this one extra API call per repo per run

[rarkins]

BTW if you configure username+password in a host rule in Renovate, it will be treated as Basic authentication

[baurmatt]

BTW if you configure username+password in a host rule in Renovate, it will be treated as Basic authentication

Interesting, thanks! 🤔 Wasn't aware that Docker registry uses basic auth as authentication method.

[saibotk]

When defining the hostRule mentioned in the description, sadly it destroys normal docker.io lookups, where it tries to connect via empty username / password credentials.

It also overwrites the default token for gitlab itself, when using a the gitlab host with a port eg. git.example.com:5000. It will then add the basic auth for both git.example.com:5000 and git.example.com.

[rarkins]

@saibotk in the description it has hostType=docker, so it should not overwrite the default token for non-docker requests. You are right though that it would apply to all ports, so you could restrict this by using matchHost=https://git.example.com/

[saibotk]

@saibotk in the description it has hostType=docker, so it should not overwrite the default token for non-docker requests. You are right though that it would apply to all ports, so you could restrict this by using matchHost=https://git.example.com/

Right it does not, that is correct. Having this rule respect ports would be another issue, yet it still does not work correctly and causes regular docker.io lookups to fail:

"message": "Response code 401 (Unauthorized)",
         "stack": "HTTPError: Response code 401 (Unauthorized)\n    at Request.<anonymous> (/usr/src/app/node_modules/got/dist/source/as-promise/index.js:118:42)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/32.208.0 (https://github.com/renovatebot/renovate)",
             "authorization": "***********",
             "accept": "application/json",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://auth.docker.io/token?service=registry.docker.io&scope=repository:gitlab/gitlab-runner:pull",
           "hostType": "docker",
           "username": "",
           "password": "",
           "method": "GET",
           "http2": false
         },
         "response": {
           "statusCode": 401,
           "statusMessage": "Unauthorized",
           "body": {"details": "incorrect username or password"},

[rarkins]

I believe the rule does support ports, for example using the syntax I gave

[saibotk]

While it does support ports, as mentioned in my example, i believe it wrongly adds authentication rules for both the domain + port and domain, and yes i already use matchHost or could you give an example on how to configure docker credentials only for a specific domain + port configuration with GitLab on the same domain (but not port). Currently, this will error, since it tries to use the wrong credentials.

[MalteMagnussen]

How do I add auth to a private GitLab Self Hosted container registry? We are running it in Kubernetes as a CronJob We have this setting on: https://docs.renovatebot.com/self-hosted-configuration/#detecthostrulesfromenv Here are the renovate docs for gitlab:

We have these secret env variables:

kind: Secret
metadata:
  name: renovate-env
data:
  GITLAB_EXAMPLE_COM_PASSWORD: ++++++++
  GITLAB_EXAMPLE_COM_TOKEN: ++++++++
  RENOVATE_TOKEN: ++++++++

and these env variables

            - env:
                - name: LOG_LEVEL
                  value: debug
                - name: RENOVATE_AUTODISCOVER
                  value: 'false'
                - name: RENOVATE_PLATFORM
                  value: gitlab
                - name: RENOVATE_ENDPOINT
                  value: 'https://gitlab.example.com/api/v4/'
                - name: RENOVATE_GIT_AUTHOR
                  value: Renovate Bot <[email protected]>
                - name: RENOVATE_REPOSITORIES
                  value: >-
                    ["xxx","yyy","zzz"]
                - name: RENOVATE_DETECT_HOST_RULES_FROM_ENV
                  value: 'true'
                - name: GOPRIVATE
                  value: gitlab.example.com
                - name: GITLAB_EXAMPLE_COM_USERNAME
                  value: renovate-bot
              envFrom:
                - secretRef:
                    name: renovate-env

Logs from run, showing forbidden 403 on the request: renovate-bot (14).txt

The docs are not very clear on how to set this up imo. It is difficult to figure out how to name and set the env variables to get the right hostRules config.

---

EDIT: Found the Solution

If I prefix with DOCKER_ it works.

                - name: DOCKER_GITLAB_EXAMPLE_COM_USERNAME
                  value: renovate-bot

  DOCKER_GITLAB_EXAMPLE_COM_PASSWORD: ++++++++

[MalteMagnussen]

OP mentions we have to use "hostType", but the documentation says different? https://docs.renovatebot.com/modules/platform/gitlab/

Set the RENOVATE_HOST_RULES CI variable to [{"matchHost": "${CI_REGISTRY}","username": "${GITLAB_USER_NAME}","password": "${RENOVATE_TOKEN}"}].

  "hostRules": [
    {
      "description": "Personal token of the Gitlab renovate user with read_registry scope only.",
      "hostType": "docker",
      "matchHost": "registry.example.de",
      "username": "renovate-read-registry",
      "password": "glpat-xxxxxxxxxx"
    }
  ],

[rarkins]

Docs are in this repo, PRs welcome

[MalteMagnussen]

@rarkins - Well, that was my first try, but the edit button returns 404: https://docs.renovatebot.com/modules/platform/gitlab/

https://github.com/renovatebot/renovate/edit/main/docs/usage/modules/platform/gitlab/index.md

Btw, thanks for creating Renovate Bot. Our Developers are so excited to get automated dependency updates. They love it.

[rarkins]

Hi @MalteMagnussen, glad to hear it, and here's a direct link to the source .md file: https://github.com/renovatebot/renovate/blob/main/lib/modules/platform/gitlab/index.md

I think the edit button gets it wrong because this particular page is semi-autogenerated from source instead of docs/. @HonkingGoose do you know if we can fix that manually?

[HonkingGoose]

@rarkins I think this issue has the information we need:

• https://github.com/renovatebot/renovate/issues/19309

If I understand correctly we need to put some kind of override into the Markdown files or the code.

[stefan-lange]

Working setup: I have a self-hosted renovate-bot in kubernetes at home and he checks private gitlab projects hosted on gitlab.com (container registry: registry.gitlab.com). The bot has its own user with access to the projects.

Based on @MalteMagnussen's solution, the following config snippet works for me. It is part of a kubernetes secret which will be converted to environment variables via envFrom/secretRef.

  RENOVATE_PLATFORM: 'gitlab'
  RENOVATE_ENDPOINT: 'https://gitlab.com/api/v4'
  RENOVATE_GIT_AUTHOR: 'Botan Botinsky <[email protected]>'
  # gitlab-scopes: read_user, api, write_repository and read_registry
  RENOVATE_TOKEN: &renovate-token 'glpat-***'
  # use custom auth for private gitlab registries
  RENOVATE_DETECT_HOST_RULES_FROM_ENV: 'true'
  DOCKER_REGISTRY_GITLAB_COM_USERNAME: 'one-bot-to-renovate-them-all'
  DOCKER_REGISTRY_GITLAB_COM_PASSWORD: *renovate-token

[eFAILution]

I have been running around in circles trying to get this to work in a self-hosted instance and @stefan-lange saves the day! None of these env vars are documented and should be provided in the self-hosted/private gitlab examples.

[HonkingGoose]

I'll be happy to help review any new docs contributions, but I don't know enough to add the documentation myself. So if you want, please open a PR to update our docs! :wink:

[baurmatt]

On Premise Helm Chart configuration needed:

  extraEnvVars:
    - name: RENOVATE_DETECT_HOST_RULES_FROM_ENV
      value: 'true'
    - name: DOCKER_REGISTRY_EXAMPLE_DE_USERNAME
      value: renovate-bot
    - name: DOCKER_REGISTRY_EXAMPLE_DE_PASSWORD
      valueFrom:
        secretKeyRef:
          key: mendRnvGitlabPat
          name: mend-renovate-ce

[baurmatt]

This has been implemented for Github with:

• https://github.com/renovatebot/renovate/issues/17705
• https://github.com/renovatebot/renovate/pull/25214

[viceice]

This has been implemented for Github with:

• https://github.com/renovatebot/renovate/issues/17705
• https://github.com/renovatebot/renovate/pull/25214

• https://github.com/renovatebot/renovate/pull/25361

is now opt-in ☝️

[douglasparker]

You’re on the wrong platform if you don’t want to be spoken to.

Douglas Parker Software Engineer

On Thu, Apr 11, 2024 at 12:25 AM github-actions[bot] < @.***> wrote:

Hi there,

Please do not unnecessarily @ mention maintainers like @rarkins or @viceice. Doing so causes annoying mobile notifications and makes it harder to maintain this repository.

For example, never @ mention a maintainer when you are creating a discussion if your desire is to get attention. This is rude behavior, just like shouting out your coffee order in a Starbucks before it's your turn.

It's OK to comment in an issue or discussion after multiple days or weeks. But please, still don't @ mention people. The maintainers try to answer most discussions, but they can't answer all discussions. If you're still not getting an answer, take a look at the information you've given us and see if you can improve it.

Thanks, the Renovate team

— Reply to this email directly, view it on GitHub https://github.com/renovatebot/renovate/issues/17940#issuecomment-2049080103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELEDLQDEUAOD5S3FTDXMLDY4Y3FXAVCNFSM6AAAAAAQUCF7NSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBZGA4DAMJQGM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[douglasparker]

Yikes, these guys seem hostile. Sorry about that, this isn’t normal behavior on GitHub.

Douglas Parker Software Engineer

On Thu, Apr 11, 2024 at 12:32 AM Matthias Baur @.***> wrote:

I wasn't even aware that you are a maintainer 🙄 But sure, blame me publicly. Good way to interact with the community!

— Reply to this email directly, view it on GitHub https://github.com/renovatebot/renovate/issues/17940#issuecomment-2049100289, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELEDLR4YFVANBVQPRXCKMLY4Y37NAVCNFSM6AAAAAAQUCF7NSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBZGEYDAMRYHE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[mbrodala]

Thanks a lot. This also works with GitLab deploy tokens (set using CI/CD variables):

DOCKER_REGISTRY_GITLAB_COM_USERNAME=gitlab+deploy-token-...
DOCKER_REGISTRY_GITLAB_COM_PASSWORD=gldt-...

The apparently most important DEBUG log lines which indicate that everything is working:

{...,"msg":"Adding password authentication for registry.gitlab.com (hostType=docker) to hostRules",...}
{...","msg":"hostRules: basic auth for https://registry.gitlab.com",...}