[Bug] Git-submodule checkout om Gitlab stopped working with 32.110.2

[rndmh3ro]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

32.111.0

Please select which platform you are using if self-hosting.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of the platform you run.

GitLab Enterprise Edition 15.1.2-ee

Was this something which used to work for you, and then stopped?

It used to work, and then stopped

Describe the bug

Git submodule extraction worked with version 32.110.1:

INFO: Repository started (repository=foo)
       "renovateVersion": "32.110.1"
 INFO: Dependency extraction complete (repository=foo)
       "baseBranch": "master",
       "stats": {
         "managers": {
           "dockerfile": {"fileCount": 1, "depCount": 2},
           "git-submodules": {"fileCount": 1, "depCount": 35},
           "gitlabci": {"fileCount": 2, "depCount": 8},
           "pip_requirements": {"fileCount": 1, "depCount": 5}
         },
         "total": {"fileCount": 5, "depCount": 50}
       }
 INFO: Repository finished (repository=foo)
       "durationMs": 35989

With version 32.110.0 (and 32.110.2) it fails (everytime and reproducably):

INFO: Repository started (repository=foo)
       "renovateVersion": "32.111.0"
 WARN: Error mapping git submodules during extraction (repository=foo)
       "err": {
         "task": {
           "commands": [
             "ls-remote",
             "--symref",
             "https://**redacted**@example.com/foo/bar/hosts.git",
             "HEAD"
           ],
           "format": "utf-8",
           "parser": "[function]"
         },
         "message": "fatal: could not read Password for 'https://**redacted**@example.com'/: No such device or address\n",
         "stack": "Error: fatal: could not read Password for 'https://**redacted**@example.com'/: No such device or address\n\n    at Object.action (/usr/src/app/node_modules/simple-git/src/lib/plugins/error-detection.plugin.ts:38:28)\n    at PluginStore.exec (/usr/src/app/node_modules/simple-git/src/lib/plugins/plugin-store.ts:24:29)\n    at /usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:114:40\n    at new Promise (<anonymous>)\n    at GitExecutorChain.handleTaskData (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:111:14)\n    at GitExecutorChain.<anonymous> (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:88:40)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/app/node_modules/simple-git/dist/cjs/index.js:55:24)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)"
       }

Changes between the versions: https://github.com/renovatebot/renovate/compare/32.110.1...32.111.1 I guess the reason is the update of the git-parse-url module: https://github.com/renovatebot/renovate/commit/210695c1a0580c18d6a2be81cabc0fce2984c856 And maybe this issue has something to do with it: https://github.com/IonicaBizau/git-url-parse/issues/140

Relevant debug logs

When running renovate on the cli with debug-log activated, I get the following output with a password-prompt at the end. This does not happen with 32.110.1.

Logs

 INFO: Repository started (repository=foo)
       "renovateVersion": "32.111.1"

DEBUG: Repo is onboarded (repository=foo)
DEBUG: Found renovate.json config file (repository=foo)
DEBUG: Repository config (repository=foo)
       "fileName": "renovate.json",
       "config": {
         "$schema": "https://docs.renovatebot.com/renovate-schema.json",
         "extends": ["config:base"],
         "git-submodules": {"enabled": true},
         "packageRules": [
           {
             "matchManagers": ["git-submodules"],
             "groupName": "ansible_submodules",
             "automerge": true
           }
         ]
       }


Password for 'https://[email protected]':

Have you created a minimal reproduction repository?

No reproduction repository

[github-actions[bot]]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[AlexDLSy]

Hello, I am encountering the same issue with Renovate 32.111.1, 32.118.0 and 32.119.1. I have exactly the same error logs.

Using Renovate self-hosted on GitLab Community 15.0.2 self-hosted.

Renovate used to work, then stopped working after updating to 32.111.1 from 32.7.9 (had not been updating for a while).

I think that it will be hard to provide a minimal reproduction with the same environment, as all is self-hosted and the repositories are private and confidential (at least in my case). Anyway the **redacted** part of the parsed URL seems weird to me and might hint to the root cause of the issue.

[AlexDLSy]

@rndmh3ro Just checking out a hinch : did the bug begin when you first updated Renovate after updating to GitLab > v15.0 ?

[astellingwerf]

That's unlikely, @AlexDLSy. I see the same behavior against GitLab 14.9, and only with versions of Renovate that include 210695c1a0580c18d6a2be81cabc0fce2984c856.

[AlexDLSy]

Ah I see, thank you for the clarification. So is it confirmed that this specific commit introduces a regression regarding GitLab submodules ?

[albertkoch]

We've also seen this issue. I can reliably reproduce by switching between 32.110.1 (working) and 32.110.2 (failing). This is with keeping the rest of the environment identical. All versions we've tried since 32.110.2 have had the same issue.

We are running renovate in a GitLab runner in a self-hosted GitLab instance.

Edit: Added missing digit from version numbers.

[eanveden]

@albertkoch I am seeing the same thing. The diff is pretty small between the two versions, just the one commit: https://github.com/renovatebot/renovate/commit/74766c1c66a9cc52c43b1f8b8a2eda1ddacf0eed

[rndmh3ro]

@albertkoch has probably mixed up the version numbers here, @eanveden. The error was introduced in 32.110.2. The changes are also linked in the first post.

[albertkoch]

@albertkoch has probably mixed up the version numbers here, @eanveden.

It looks like I'd dropped a digit in the version numbers I listed. I've updated my comment with the correct version numbers. Apologies for the confusion.

[astellingwerf]

The change that breaks this (in GitUrlParse) is that the PAT is now stored on its own in the password field, whereas this was previously stored as part of the user field, as "oauth2:{pat}". The toString function does include the user or token field, but not the password.

As the password gets dropped, the git-submodules manager will try to get credentials with the host rules. For submodules local to the platform, there are credentials available that may work, but those are registered with the hostType set to the platform ID. Because the search rule for credentials does not provide a host type, those credentials are ignored.

The workaround for the git-modules update resolution is to run with a config.js that contains a snippet like this:

  hostRules: [
    {
      username: 'oauth2',
      password: 'glpat-Go-cR2UhBo4ELbWy',
      matchHost: 'gitlab.com'
    }
  ]

Unfortunately, this workaround has crucial side effect: The search for host rules to read data from gitlab over HTTP now returns an object with username, password and token. The implementation of applyHostRules gives preference to username and password over a token. Trying to set it as a token instead of as username and password also doesn't work.

I've been looking at solutions, but I don't see an obvious one. It seems that the new implementation of GitUrlParse makes it difficult to correctly authenticate.

Steps to reproduce

This bug stems from the fact that the submodule is private, and therefore it is impossible to create a repro environment that can be shared with the Renovate team. The steps to reproduce are relatively simple though:

1. Create a private GitLab project "submodule-project", and make sure it contains at least one commit.
2. Create another sibling project "renovate-issue-16561" in GitLab.
3. Clone the project.
4. Run git submodule add ../submodule-project.
5. Create a renovate.json file with:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:base"
  ],
  "git-submodules": {
    "enabled": true
  }
}

6. Commit the contents of the "renovate-issue-16561".
7. Create and push another revision on the default branch of "submodule-project", so that the "renovate-issue-16561" project now needs renovation.
8. Run Renovate against "renovate-issue-16561".

I did not test this scenario on GitHub, but I assume that would (not) work the same way.

[rndmh3ro]

@astellingwerf what is the reason you deleted your comment with the workaround? For me it works.

[astellingwerf]

@rndmh3ro, because it still didn't work for me when I ran it in the actual GitLab pipeline. It only seemed to have an effect when run locally.

[rndmh3ro]

It did work for me when I set it on the global level in the .gitlab-ci.yml:

# workaround for https://github.com/renovatebot/renovate/issues/16561#issuecomment-1206897658
variables:
  GIT__REFS_GITLAB_INTRA_NET_USERNAME: oauth2
  GIT__REFS_GITLAB_INTRA_NET_PASSWORD: $RENOVATE_TOKEN

[viceice]

It did work for me when I set it on the global level in the .gitlab-ci.yml:

# workaround for https://github.com/renovatebot/renovate/issues/16561#issuecomment-1206897658
variables:
  GIT__REFS_GITLAB_INTRA_NET_USERNAME: oauth2
  GIT__REFS_GITLAB_INTRA_NET_PASSWORD: $RENOVATE_TOKEN

This only works when enable detectHostRulesFromEnv^1 and is effectively the same as

 hostRules: [
    {
      username: 'oauth2',
      password: process.env.RENOVATE_TOKEN,
      matchHost: 'gitlab.intra.net',
      hostType: 'git-refs'
    }
  ]

[github-actions[bot]]

When a bug has been marked as needing a reproduction, it means nobody can work on it until one is provided. In cases where no reproduction is possible, or the issue creator does not have the time to reproduce, we unfortunately need to close such issues as they are non-actionable and serve no benefit by remaining open. This issue will be closed after 7 days of inactivity.

[rndmh3ro]

There is a reproduction method here: https://github.com/renovatebot/renovate/issues/16561#issuecomment-1206854917

[viceice]

@rndmh3ro That's not a reproduction, checkout the first comment for a descriptions what we need

[dersvenhesse]

I suppose I managed to reproduce the issue with some small projects on GitLab.

There is

• a demo project https://gitlab.com/renovate-16561/demo
• private submodule https://gitlab.com/renovate-16561/submodule

The latter is included in the demo project and contains only a README:

The demo project has RENOVATE_EXTRA_FLAGS set to --autodiscover=true via CI/CD variables as well as an RENOVATE_TOKEN with access to the private submodule project. In renovate.json the handling of submodules is enabled via git-submodules": { "enabled": true }.

When running the pipeline with renovate-runner/v8.144.0 (using renovate v32.110.1) everything works fine (https://gitlab.com/renovate-16561/demo/-/jobs/2920447823) and the pipeline finishes successfully.

After updating the renovate-runner to the current version v8.204.0 the issue occurs (https://gitlab.com/renovate-16561/demo/-/jobs/2920487724):

I hope this helps.

[viceice]

ok, debugging against : https://gitlab.com/viceice-test/with-submodule

the host rule doesn't match, very strange

[viceice]

I've managed to debug this, see linked PR

[renovate-release]

:tada: This issue has been resolved in version 32.174.2 :tada:

The release is available on:

• GitHub release
• 32.174.2

Your semantic-release bot :package::rocket:

[dersvenhesse]

Unfortunately, for me and my demo project the issue seems to stay with 32.174.2 (https://gitlab.com/renovate-16561/demo/-/jobs/2931769263):

[viceice]

@dersvenhesse Do you configure any hostrule?

For this to work you need a host-rule like this:

    {
      matchHost: 'gitlab.com',
      token: 'xxxxxxx'
    }

This is because the default renovate generated host-rule for platform includes a hostType and the sub-modules manager doesn't send any hostType, so the deafult rule doesn't match and then the token is empty.

https://github.com/renovatebot/renovate/blob/bf378a46377cd98bb00e7b4053c179474316ce27/lib/modules/manager/git-submodules/extract.ts#L118

https://github.com/renovatebot/renovate/blob/bf378a46377cd98bb00e7b4053c179474316ce27/lib/util/git/url.ts#L41-L59

[viceice]

@rarkins maybe we should detect platform in sub-modules manager and fall-back to git-refs hostType ?

[dersvenhesse]

@viceice, thanks for the hint. Everything works fine with the setted host-rule. 🚀

[viceice]

ok, i've preparing a new PR to fix that, so it will use the default host rule

[jimbojd72]

Anything missing to merge the fix?

[AlexDLSy]

ok, i've preparing a new PR to fix that, so it will use the default host rule

@viceice Does it mean that after your PR is approved and merged we won't need to use workarounds such as described above ? If those were still to be needed, what is the official workaround ?

[viceice]

maybe

[SeaLife]

Hey, just a small heads-up, this seem to be still an issue.. Creating a config.js with the HostRule from: https://github.com/renovatebot/renovate/issues/16561#issuecomment-1207955543 fixed my issue. :)