ValueError: Problem accessing HEAD branch

[fmaussion]

Hi! After updating my images today, nbgitpuller started to throw this error when starting from an url which worked previously.

Here is the url (generated with the generator) and the error message:

https://mybinder.org/v2/gh/OGGM/binder/stable?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FOGGM%252Ftutorials%26urlpath%3Dlab%252Ftree%252Ftutorials%252F%26branch%3Dstable

Traceback (most recent call last):
File "/srv/conda/envs/notebook/lib/python3.9/site-packages/nbgitpuller/pull.py", line 120, in resolve_default_branch
    head_branch = subprocess.run(
File "/srv/conda/envs/notebook/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'ls-remote', '--symref', '--', 'https://github.com/OGGM/tutorials', 'HEAD']' returned non-zero exit status 128.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/conda/envs/notebook/lib/python3.9/site-packages/nbgitpuller/handlers.py", line 76, in get
    gp = GitPuller(repo, repo_dir, branch=branch, depth=depth, parent=self.settings['nbapp'])
File "/srv/conda/envs/notebook/lib/python3.9/site-packages/nbgitpuller/pull.py", line 76, in __init__
    self.branch_name = self.resolve_default_branch()
File "/srv/conda/envs/notebook/lib/python3.9/site-packages/nbgitpuller/pull.py", line 136, in resolve_default_branch
    raise ValueError(m)
ValueError: Problem accessing HEAD branch: https://github.com/OGGM/tutorials

[fmaussion]

After looking into this a bit more, it turns out that the machines we now receive from mybinder ask for a password when one runs the command:

git ls-remote --symref -- https://github.com/OGGM/tutorials HEAD

Which is not the case when I run the command locally. Is this some change on the github side, or some git configuration issue?

[consideRatio]

Does it make a difference if you do...

git ls-remote --symref https://github.com/OGGM/tutorials HEAD

instead?

The upgrade you got took you from 0.10.1 to 0.10.2 I assume, and the only difference is https://github.com/jupyterhub/nbgitpuller/commit/f25d3f2685035c11bd668d48e71caf4fc245ba68 related to these -- in between that was added for security reasons.

[fmaussion]

Thanks @consideRatio

No, it also asks a password. But now I wonder if the problem isn't related to an update to nbgitpuller, but rather to an update in the git config or linux images used by mybinder?

[fmaussion]

To reproduce, open a terminal in https://mybinder.org/v2/gh/OGGM/binder/stable and run the above commands. Here in screenrecording:

https://user-images.githubusercontent.com/10050469/131307279-9e5425e8-f55e-4a4b-8c75-7e54db21763a.mp4

[consideRatio]

@fmaussion ah you were asked for credentials no matter what it seems, that is good because then that change about -- seems unrelated which I hoped it was - because we can't just revert that security fix.

I observe that I could also run git ls-remote --symref -- https://github.com/OGGM/tutorials HEAD locally when having git configured with me as an author etc. I'm not sure at all why this happens yet but I think you have pinpointed a clear reproducible situation to look into! Nice work!

[consideRatio]

After updating my images today,

Do you know what version of nbgitpuller you had before?

[fmaussion]

Do you know what version of nbgitpuller you had before?

I was using a tagged image from 2021 05 16

You can still open it on binder with:

https://mybinder.org/v2/gh/OGGM/binder/5959f23f1cdcff95384bbe0897b8290150969b2f

In this image, the git ls-remote --symref -- https://github.com/OGGM/tutorials HEAD returns properly. (i.e. nothing to do with gitpuller).

[TimoRoth]

Immediate observation from my side: This only happens if you're currently in ~, which is a git repo in that new image. If I cd to / before running the ls-remote command(or rm -rf .git), it runs just fine without prompting for a password.

~ being a git repo is also the case in the old image, but for some reason it does not bother git(which is the same version in both images) there.

[fmaussion]

Thanks @TimoRoth - an immediate fix for us would be to delete ~/.git in repo2docker's PostBuild script, but maybe this is something nbgitpuller needs to address still?

[TimoRoth]

Ok, found what's causing it: The new image has

[http "https://github.com/"]
        extraheader = AUTHORIZATION: basic <censored token>

In .git/config. So while in that git repo, git will pass that info along. And it's apparently invalid, so github asks you to fix your auth, resulting in git presenting a login prompt.

[fmaussion]

Thanks - excellent. Is this something repo2docker should fix? Ping @yuvipanda just in case.

[consideRatio]

@TimoRoth wait what image has that and why? This is potentially sensitive credentials that shouldn't be embedded in an image, not by binderhub, mybinder.org, or an end user letting a repo be built with mybinder.org

[manics]

~Maybe it's due to the switch to building the Dockerfile ourselves?~ ~https://github.com/jupyterhub/repo2docker/blob/6e2a6af959a366ea9cd5e268450122d0f7064afd/Dockerfile#L7~

Edit: Ignore this, it's the container used for running the build not the output. In any case find / -name .git returns nothing.

[fmaussion]

@consideRatio for the context:

• we build the images using repo2docker: https://github.com/OGGM/r2d - therefore the .git folder in HOME is meant for this repository.
• we then use nbgitpuller to pull the content from another (any) repository.

I don't know if this is relevant, but we actually use CI to build the images in https://github.com/OGGM/r2d, and then pull them from dockerhub in https://github.com/OGGM/binder

[manics]

@fmaussion If you pass https://github.com/OGGM/r2d to mybinder.org (instead of https://github.com/OGGM/binder) and set nbgitpuller to pull do you still have a problem?

[fmaussion]

@manics no I don't - this therefore has to do with the tokens being for the r2d repository and then the use of the dockerfile trick in the reproducible repository: https://github.com/OGGM/binder/blob/master/binder/Dockerfile

I'm not sure anymore if this is something you need to address, as we are misusing repo2docker a little bit here (note that this used to work a few months ago still).

Context for our reasons of having two repositories: https://discourse.jupyter.org/t/reproducible-binder-environments-with-repo2docker-dockerhub-and-nbgitpuller

[manics]

I understand what you're doing, it's a good solution and it's not misusing repo2docker. In case you weren't aware there's a GitHub action that supports your pre-build use-case: https://github.com/jupyterhub/repo2docker-action

I asked about passing https://github.com/OGGM/r2d to mybinder.org to help narrow down the problem as there are several components (mybinder, repo2docker, nbgitpuller, your notebook repo, your docker environment repo, your CI build).

note that this used to work a few months ago still

Your GitHub workflow was only added 2 days ago (https://github.com/OGGM/r2d/commits/master/.github), how were you building the image before? If you go back to the old process does everything work again?

[fmaussion]

Your GitHub workflow was only added 2 days ago

Oh right.

how were you building the image before?

With Travis-CI! @TimoRoth can comment more on the rest. I didn't think that this change would make a difference :see_no_evil:

[fmaussion]

I am making changes in the repos to delete the .git folder in postBuild.

For future reference, here are the permanent mybinder builds to check what happened:

• https://mybinder.org/v2/gh/OGGM/binder/5959f23f1cdcff95384bbe0897b8290150969b2f : a 3-months build with Dockerfile + Travis-CI which worked
• https://mybinder.org/v2/gh/OGGM/binder/4a7dbde0daf0698d39116bc99ef9bd25610a59de : a recent build after we switched to gh-actions instead of travis to build the base image (if this is what is causing the issue)
• https://mybinder.org/v2/gh/OGGM/r2d/8f770cd574c6726913a6bdb1dc785e7cfa7460cb : a build without the use of CI, or the second repository and the dockerfile, which seems to work as expected.