Renovate cannot clone/update submodule in private repo: prompts for password

[okainov]

How are you running Renovate?

• [ ] WhiteSource Renovate hosted app on github.com
• [x] Self hosted

Please select which platform you are using:

• [x] github.com

Renovate version: docker latest

Describe the bug

• Having the repo in private org with submodule to the very same private org
• Both repos are accessible with the very same token
• Run renovate docker run -v /tmp/renovate/config.js:/usr/src/app/config.js -it --rm -e LOG_LEVEL=debug renovate/renovate
• Make sure it will prompt for password - not expected, I expect it should use token

Relevant debug logs

logs:

 INFO: Repository started (repository=myorg/myrepo)
       "renovateVersion": "25.38.2"
DEBUG: Using localDir: /tmp/renovate/repos/github/myorg/myrepo (repository=myorg/myrepo)
DEBUG: initRepo("myorg/myrepo") (repository=myorg/myrepo)
DEBUG: Overriding default GitHub endpoint (repository=myorg/myrepo)
       "endpoint": "https://api.github.com/"
DEBUG: myorg/myrepo default branch = master (repository=myorg/myrepo)
DEBUG: Using personal access token for git init (repository=myorg/myrepo)
DEBUG: resetMemCache() (repository=myorg/myrepo)
DEBUG: Resetting npmrc (repository=myorg/myrepo)
DEBUG: checkOnboarding() (repository=myorg/myrepo)
DEBUG: isOnboarded() (repository=myorg/myrepo)
DEBUG: findFile(renovate.json) (repository=myorg/myrepo)
DEBUG: Initializing git repository into /tmp/renovate/repos/github/myorg/myrepo (repository=myorg/myrepo)
DEBUG: git clone completed (repository=myorg/myrepo)
       "durationMs": 6311
DEBUG: Cloning git submodule at submodule-name(repository=myorg/myrepo)
Username for 'https://github.com':

Seems like there was something implemented in https://github.com/renovatebot/renovate/issues/5471, but I don't see how it works... Neither we use SSH, everything is on HTTPS anyway.

config.js:

module.exports = {
  token: 'mytoken',
  platform: 'github',
  logLevel: 'debug',
  onboardingConfig: {
    extends: ['config:base'],
  },
  addLabels: ["deps"],
  labels: ["renovate"],
  repositories: ['myorg/myrepo'],
  renovateFork: false,
  gitAuthor: "Bot <[email protected]>",
  username: "myname",
  onboarding: true,
  printConfig: true,
  requireConfig: false,
};

renovate.json in the repo

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:base", "docker:enableMajor"], 
  
  "enabledManagers": ["git-submodules"],
  
  "git-submodules": {
    "enabled": true
  }
}

Additional context

...

[rarkins]

What does the .gitmodules file contain?

[okainov]

@rarkins

[submodule "submodule-name"]
        path = submodule-name
        url = https://github.com/myorg-2/submodule-name

(okay, it's different private org, but it is still accessible with the same token)

[rarkins]

It's definitely undesirable if it's "hanging" while asking for a username. @JamieMagee we should try to fix that if we can, e.g. is there a non-interactive flag which will fail fast?

@okainov in the meantime you'll need to add cloneSubmodules: false to this repo or all similar repos to skip the submodule clone step.

[okainov]

@rarkins just tried to add this to renovate.json - didn't make any effect. Tried to add into config.js - something has changed, but it still prompts for password (but seems like in another step):

log:

DEBUG: processRepo() (repository=myorg/myrepo)
DEBUG: No baseBranches (repository=myorg/myrepo)
DEBUG: extract() (repository=myorg/myrepo)
DEBUG: Setting current branch to master (repository=myorg/myrepo)
DEBUG: latest commit (repository=myorg/myrepo)
       "branchName": "master",
       "latestCommitDate": "2021-06-09T13:24:08+02:00"
DEBUG: Applying enabledManagers filtering (repository=myorg/myrepo)
DEBUG: Using file match: (^|/).gitmodules$ for manager git-submodules (repository=myorg/myrepo)
DEBUG: Matched 1 file(s) for manager git-submodules: .gitmodules (repository=myorg/myrepo)
Username for 'https://github.com':

[rarkins]

That's failing at a different step now - the git-submodules manager. FYI we don't support private submodules right now and the feature request for it is #10149

I would like to leave this open as a bug report though, with the intention to:

• Clearly document the limitation (maybe easiest in the git-submodules manager readme)
• Try to fail-fast instead of prompting for username

[viceice]

I think the problem is the checkout. We don't have any auth configured for private submodules.

[okainov]

@rarkins @viceice but...but... wasn't #5471 about exactly this? And #8164 was clearly saying A token is correctly applied, as configured in hostrules... I was really expecting this should work...

[rarkins]

@okainov I'm not sure exactly what's going wrong, but without a reproduction then we need to declare it as unsupported for now because multiple people have reported it as not working. You're welcome to also debug it locally to step through the logic privately without needing to share anything.

[viceice]

@rarkins @viceice but...but... wasn't #5471 about exactly this? And #8164 was clearly saying A token is correctly applied, as configured in hostrules... I was really expecting this should work...

If you read the pr description you'll see the private submodule is configured via ssh. So I'm sure @JamieMagee used ssh key agent for initial submodule checkout.

His pr only apply the token for the manager for the submodule update. the initial checkout is done by renovate on repo clone, which doesn't configure any token for submodules.

[okainov]

If you read the pr description you'll see the private submodule is configured via ssh. So I'm sure @JamieMagee used ssh key agent for initial submodule checkout.

I'm not a JS expert, but reading the code it looked really like "read SSH url, transform it to HTTPS, add token, clone as HTTPS".... And that's also what spec-"test" was verifying, I believe....

[viceice]

But as i said, renovate initialize the git submodules before the sub-module manager extracts something. https://github.com/renovatebot/renovate/blob/9f2e761b4ff06072e9703b184bb7688501440e52/lib/util/git/index.ts#L321

So there is no auth config for submodules, so it fails for private sub-modules.

[JamieMagee]

@viceice is correct. We do not apply any auth tokens at initial clone of submodules, only when checking for updates. Submodules are cloned here and updated here. But before we update submodules, we convert the URL from SSH to HTTPS and apply and hostrule tokens.

[okainov]

I've tried to investigate this a little bit, what I found so far:

• Passing token to submodule is possible from git-js library, see here https://github.com/steveukx/git-js/issues/635. ~So from the current renovate implementation it would mean to get the token and add it to git object initialization, seems relatively easy even for non-JS/TS developers...~ But unfortunately Github does not allow to clone repos like that, so it's not an option.
• Change from #8164 is really-really misleading because it does not use "token" from config. It really looks inside specific "host rules". As a user I find this incredibly inconvenient, because whole Renovatebot works with simple "token" but this part of functionality does not work. config.js:

module.exports = {
  token: 'mytoken',      // That's what I had before, what you could see in the first post and it worked well... but not for the functionality from #8164
  "hostRules": [
    {    // This block is what I had to add in order to use logic from #8164, mind *the very same* token
      "matchHost": "github.com",
      "token": 'mytoken'
    }
  ],

• As git-js contributors mentioned, git-js is capable of using .netrc file implicitly, I've actually tried that (mount my .netrc into renovate docker container) and it worked like a charm and I got my submodule PR created. Even when I removed "hostrules" it still worked well.

[asgoel]

I'm having a similar issue, however I think we need to clone our submodule because it is a python dependency in our main project. We're using the Github app version of renovate though, is there something specific we can do to give the renovate bot access to our submodule repo? I tried adding renovate-bot as a collaborator on the submodule project, but I don't think it worked.

[viceice]

@asgoel renovate app needs to be enabled on submodule repo too, then it should be able to it checkout

[asgoel]

@viceice that didn't seem to work for us unfortunately. Still seeing an error where the submodule cannot be checked out.

[asgoel]

@viceice is there anywhere we can reach out for github specific support? or is this the best avenue

[cguldner]

• As git-js contributors mentioned, git-js is capable of using .netrc file implicitly, I've actually tried that (mount my .netrc into renovate docker container) and it worked like a charm and I got my submodule PR created. Even when I removed "hostrules" it still worked well.

Using a .netrc in my container worked as well, thank you!