git-lfs icon indicating copy to clipboard operation
git-lfs copied to clipboard

Published 20 hours ago • verified publisher icon git-lfs

Running into what appears to be a fairly common issue with git-lfs failing to authenticate during push (verify locks)

Open haikusw opened this issue 2 years ago • 22 comments

Reporting as a bug but not 100% sure what the root cause is: user issue, a git-lfs assumption/spec interpretation that GitHub server isn't agreeing with, a bug in git-lfs, false assumption about user systems, … ¯_(ツ)_/¯

I'm inclined to think user issue, but I can't figure out what it is. If nothing else, there's some documentation that needs writing (I've read everything I could find in the GitHub.com documents and elsewhere online and tried a lot of different things).

Describe the bug git lfs push <remote> <branch> fails with Authentication error 403

This issue has been reported by numerous people over the years (e.g., #2114, #2219, #1852, #2361, etc, etc), and I've read all of those I could find. Few of them have actual fixes, and the few that do don't seem to fix the issue I'm having. Sometimes the response was "GitHub server issue - go talk to GitHub support", which was surprising. Surely a GitHub dev can get a ticket filed that gets paid a lot more attention to than we do; and maybe someone familiar with the git-lfs internals can shed enough light on the specific issue to help the server devs find and fix it, especially since it's been tripping people up for at least 5 years). If nothing else, there's a documentation improvement issue (I'd be happy to write something up if I could figure out what the root cause was).

To Reproduce

  • I am attempting to use a GitHub personal access token.
  • I have a GitHub hosted read-write access fork ("haikusw_origin") of a GitHub repo to which I don't have write access ("origin").
  • This repo is using lfs.
  • I have a branch ("async") checked out and tracking a branch on the read-only repo ("origin").
  • I wish to push this branch ("async") to the fork I have read-write access to ("haikusw_origin").
  • I execute the command git lfs push haikusw_origin async
  • …and receive the following error:

error: Authentication error: Authentication required: You must have push access to verify locks

Expected behavior The branch "async" should be created in the repo "haikusw_origin"

System environment The version of your operating system, plus any relevant information about platform or configuration (e.g., container or CI usage, Cygwin, WSL, or non-Basic authentication). If relevant, include the output of git config -l as a code block.

  • macOS X 12.3.1 on an M1 MBP

  • zsh (default for macOS X 12.x)

  • git version 2.35.2

  • git-lfs version 3.1.2 (GitHub; Darwin arm 64; go 1.17.6)

  • ssh-agent is running (ssh-agent dumps SSH_AUTH_SOCK and SSH_AGENT_PID values, set | grep SSH shows them set).

  • echo "host=github.com\nprotocol=https" | git credential fill returns the expected output:

protocol=https host=github.com username=haikusw password={correct personal token}

However, when I do GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async immediately before the POST that returns 403 I see the following two log entries which I then tried on the command line directly:

% /usr/bin/security 'list-keychains'

"/Users/tyler/Library/Keychains/login.keychain-db"
 …
"/Library/Keychains/System.keychain"

% /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'gist.github.com' '/Library/Keychains/System.keychain' % (no output)

There is no output from the second one, which doesn't seem all that surprising - I don't expect user certificates will be in the System keychain (?).

I wonder if this is a clue? I see in the git-lfs source code that there is an intentional ignoring of the login keychain "to match browsers" or something like that. Maybe this is unrelated, but it did catch my eye.

Git Config:

credential.helper=osxkeychain
mergetool.keepbackup=true
user.name=Tyler
user.email={email address}
core.excludesfile=/Users/tyler/.gitignore_global
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
difftool.Kaleidoscope.cmd=ksdiff --partial-changeset --relative-path "$MERGED" -- "$LOCAL" "$REMOTE"
mergetool.Kaleidoscope.cmd=ksdiff --merge --output "$MERGED" --base "$BASE" -- "$LOCAL" --snapshot "$REMOTE" --snapshot
mergetool.Kaleidoscope.trustexitcode=true
commit.template=/Users/tyler/.stCommitMsg
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.haikusw_origin.url=https://[email protected]/haikusw/isowords.git
remote.haikusw_origin.gtserviceaccountidentifier=58ebcb006ed046c6e32cbbaad31fb77d0139ba9080ff2071fb3886dd99355400
remote.haikusw_origin.fetch=+refs/heads/*:refs/remotes/haikusw_origin/*
branch.haikusw_main.remote=haikusw_origin
branch.haikusw_main.merge=refs/heads/main
[email protected]:pointfreeco/isowords.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.main.remote=origin
branch.main.merge=refs/heads/main
branch.async.remote=origin
branch.async.merge=refs/heads/async
user.name=haikuty
user.email={email address}
lfs.repositoryformatversion=0

Output of git lfs env

Ok - I'm wondering why this output says "(auth=none)" at the end of the "haikusw_origin" endpoint? That seems incorrect and possibly an indication of the root problem? (he says hopefully…).

% git lfs env
git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
git version 2.35.2

Endpoint=https://github.com/pointfreeco/isowords.git/info/lfs (auth=none)
  [email protected]:pointfreeco/isowords.git
Endpoint (haikusw_origin)=https://[email protected]/haikusw/isowords.git/info/lfs (auth=none)
LocalWorkingDir=/Users/Shared/OpenSource/isowords
LocalGitDir=/Users/Shared/OpenSource/isowords/.git
LocalGitStorageDir=/Users/Shared/OpenSource/isowords/.git
LocalMediaDir=/Users/Shared/OpenSource/isowords/.git/lfs/objects
LocalReferenceDirs=
TempDir=/Users/Shared/OpenSource/isowords/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/Users/Shared/OpenSource/isowords/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/opt/homebrew/Cellar/git/2.35.2/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Additional context Any other relevant context about the problem here.

A different repo forked from the same organization doing the same operation (pushing a branch they have but my fork doesn't to my fork) succeeds as expected. It's only when using git-lfs that the error happens.

Checking git lfs env (though it doesn't have the lfs git hooks installed and I don't think it's using lfs), I see the same "(auth=none)" after the endpoint name. Maybe a red herring.

% git lfs env   
git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
git version 2.35.2

Endpoint=https://github.com/pointfreeco/swift-parsing.git/info/lfs (auth=none)
Endpoint (haikusw-origin)=https://[email protected]/haikusw/swift-parsing.git/info/lfs (auth=none)
LocalWorkingDir=/Users/Shared/OpenSource/PointFree-swift-parsing
LocalGitDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git
LocalGitStorageDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git
LocalMediaDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git/lfs/objects
LocalReferenceDirs=
TempDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/opt/homebrew/Cellar/git/2.35.2/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Happy to try more things if there's something useful to try. Thanks.

haikusw avatar May 10 '22 21:05 haikusw

Hey, I'm sorry you're having trouble. If I understand correctly, I wonder if this is your problem:

I have a GitHub hosted read-write access fork ("haikusw_origin") of a GitHub repo to which I don't have write access ("origin"). I wish to push this branch ("async") to the fork I have read-write access to ("haikusw_origin").

According to their documentation, GitHub says (in the "Tips" box):

In forks, bandwidth and storage usage count against the root of the repository network.... Forking and pulling a repository counts against the parent repository's bandwidth limit.

Forges like GitLab and GitHub typically store all the forks of repository together in a way that minimizes duplicate files between the forks. However, at least for GitHub, this means Git LFS usage is tracked against the parent repository. So you may not be able to push your Git LFS objects even though your fork is read/write.

I would definitely recommend getting in touch with GitHub Support as they can look into your personal situation and (with your permission) repositories, whereas this project is just for the open-source Git LFS client.

As for the (auth=none) annotation in the git lfs env output, I agree that's rather mysterious. What it means is that you haven't explicitly configured a section in your Git config that looks like, for example:

[lfs "https://[email protected]/haikusw/isowords.git/info/lfs"]
        access = basic

But typically you should not need to do that; Git LFS will infer the authorization scheme from the response it gets from the server.

chrisd8088 avatar May 10 '22 23:05 chrisd8088

Thank you for the reply. Interesting to learn about the bandwidth and storage use counting against the root of the repository network - quite unexpected and there's no way I would have figured out that was what that error message meant. Hopefully that can get improved somehow if this is root cause. I'll give support a hello and report back what they say.

Also - thanks for the clarification about "(auth=none)". Definitely sends the wrong message :) "(auth=unspecified)" or "(auth=inferred)", while longer, might be clearer.

haikusw avatar May 11 '22 00:05 haikusw

I've continued to try to investigate this from the client side, and I notice that a difference you report between the working and non-working environments is that the latter has an SSH endpoint while the former does not.

Are you able to switch to an HTTPS endpoint in your Git configuration and see if that works?

And, would you be able to provide a complete set of steps to reproduce the problem, starting from a git clone of your fork?

And as I noted, please do be mindful of the fact that fetching and adding Git LFS objects will be counted against the parent repository's usage by GitHub.

chrisd8088 avatar May 11 '22 22:05 chrisd8088

Interesting! Thank you. I should be able to do both those things later this evening and will report back.

Update on my support query: Support verified some basic facts and that it seems like it should work and then opened a ticket with engineering to investigate.

haikusw avatar May 11 '22 22:05 haikusw

During edits to my original post I seem to have lost the test for ssh connectivity that I did based on GitHub documentation:

% ssh -T [email protected]
Hi haikusw! You've successfully authenticated, but GitHub does not provide shell access.

haikusw avatar May 12 '22 03:05 haikusw

Hi @chrisd8088 - so I made a new folder and cloned the repo fresh being sure to use the https urls.

It appears that I git-lfs still get an authentication error but it's a 401 instead of a 403, and then git-lfs retried using basic repo access method and that succeeded!

I hope this gives you some clue to the what and why of this issue that people seem to run into off and on.

Here's the sequence and results (note that I did a different test immediately after this one; it's below this):

% git clone https://github.com/pointfreeco/isowords.git
...
% cd isowords

% git remote add haikusw_origin https://github.com/haikusw/isowords.git

% git remote -v
haikusw_origin	https://github.com/haikusw/isowords.git (fetch)
haikusw_origin	https://github.com/haikusw/isowords.git (push)
origin	https://github.com/pointfreeco/isowords.git (fetch)
origin	https://github.com/pointfreeco/isowords.git (push)

% git checkout async
branch 'async' set up to track 'origin/async'.
Switched to a new branch 'async'

% GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async
20:39:20.625526 git.c:745               trace: exec: git-lfs push --dry-run haikusw_origin async
20:39:20.626129 run-command.c:654       trace: run_command: git-lfs push --dry-run haikusw_origin async
20:39:20.681171 trace git-lfs: exec: git 'version'
20:39:20.686781 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
20:39:20.690735 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
20:39:20.694321 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
20:39:20.698481 trace git-lfs: exec: git 'config' '--includes' '-l'
20:39:20.701379 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
20:39:20.703902 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
20:39:20.707879 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
20:39:20.713282 trace git-lfs: Upload refs [async] to remote haikusw_origin
20:39:20.713425 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
20:39:20.717220 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
20:39:20.720169 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'haikusw_origin'
20:39:21.188744 trace git-lfs: exec: /usr/bin/security 'list-keychains'
20:39:21.219142 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'github.com' '/Library/Keychains/System.keychain'
20:39:21.237585 trace git-lfs: HTTP: POST https://github.com/haikusw/isowords.git/info/lfs/locks/verify
> POST /haikusw/isowords.git/info/lfs/locks/verify HTTP/1.1
> Host: github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}20:39:21.748874 trace git-lfs: HTTP: 401


< HTTP/2.0 401 Unauthorized
< Content-Length: 153
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 03:39:21 GMT
< X-Frame-Options: DENY
< X-Github-Request-Id: ECAC:3720:5DCDE6:C0B8F1:627C8169
< 
20:39:21.749126 trace git-lfs: HTTP: {"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"ECAC:3720:5DCDE6:C0B8F1:627C8169"}

{"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"ECAC:3720:5DCDE6:C0B8F1:627C8169"}

20:39:21.751221 trace git-lfs: setting repository access to basic
20:39:21.751411 trace git-lfs: exec: git 'config' '--includes' '--replace-all' 'lfs.https://github.com/haikusw/isowords.git/info/lfs.access' 'basic'
20:39:21.764002 trace git-lfs: api: http response indicates "basic" authentication. Resubmitting...
20:39:21.764397 trace git-lfs: creds: git credential fill ("https", "github.com", "")
20:39:21.764466 trace git-lfs: exec: git 'credential' 'fill'
20:39:21.808731 trace git-lfs: Filled credentials for https://github.com/haikusw/isowords.git
20:39:21.809149 trace git-lfs: exec: /usr/bin/security 'list-keychains'
20:39:21.822844 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'github.com' '/Library/Keychains/System.keychain'
20:39:21.836043 trace git-lfs: HTTP: POST https://github.com/haikusw/isowords.git/info/lfs/locks/verify
> POST /haikusw/isowords.git/info/lfs/locks/verify HTTP/1.1
> Host: github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}{"ref":{"name":"refs/heads/async"}}20:39:22.165743 trace git-lfs: HTTP: 200


< HTTP/2.0 200 OK
< Content-Length: 41
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 03:39:22 GMT
< X-Frame-Options: DENY
< X-Github-Request-Id: ECAD:611C:3C5467:92CD5E:627C816A
< 
20:39:22.165938 trace git-lfs: creds: git credential approve ("https", "github.com", "")
20:39:22.166057 trace git-lfs: exec: git 'credential' 'approve'
20:39:22.300295 trace git-lfs: HTTP: {"ours":[],"theirs":[],"next_cursor":""}

{"ours":[],"theirs":[],"next_cursor":""}
20:39:22.300799 trace git-lfs: tq: running as batched queue, batch size of 100
20:39:22.300915 trace git-lfs: run_command: git rev-list --objects --ignore-missing --not --remotes=haikusw_origin --stdin --
20:39:22.302014 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
20:39:22.303645 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'

push 73b2f912875308b6669232c2bad2456ae93f5aae954bc44cc2e45a0a50d34fdc => App/AppClip/Assets.xcassets/AppIcon.appiconset/[email protected]
... many lines like the previous ...

20:39:22.341846 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
20:39:22.341869 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
20:39:22.341894 trace git-lfs: filepathfilter: accepting "tmp"

I don't understand why git remote -v shows this difference that you called out in my original clone and test folder

Endpoint=https://github.com/pointfreeco/isowords.git/info/lfs (auth=none)
  [email protected]:pointfreeco/isowords.git

It's the HTTPS url but then has that SSH line below it.

Only think I can think of is that I added the remote with the Git Tower macOS git app (which I do generally use; when it started giving me errors I switched to command line to see if it was a Tower bug and ).

Out of curiosity I tried removing the haikusw_origin and then adding it back via the [email protected]… url. Different behavior with different looking failures, but ultimately recovering and completing.

Here's that sequence of commands and outputs (which begin with things in the state at the end of the above sequence/test):

% git remote remove haikusw_origin
% git remote -v
origin	https://github.com/pointfreeco/isowords.git (fetch)
origin	https://github.com/pointfreeco/isowords.git (push)

% git remote add haikusw_origin [email protected]:haikusw/isowords.git
% git remote -v
haikusw_origin	[email protected]:haikusw/isowords.git (fetch)
haikusw_origin	[email protected]:haikusw/isowords.git (push)
origin	https://github.com/pointfreeco/isowords.git (fetch)
origin	https://github.com/pointfreeco/isowords.git (push)

% GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async
20:49:25.953540 git.c:745               trace: exec: git-lfs push --dry-run haikusw_origin async
20:49:25.953948 run-command.c:654       trace: run_command: git-lfs push --dry-run haikusw_origin async
20:49:25.999396 trace git-lfs: exec: git 'version'
20:49:26.005434 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
20:49:26.011366 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
20:49:26.016236 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
20:49:26.019658 trace git-lfs: exec: git 'config' '--includes' '-l'
20:49:26.022775 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
20:49:26.025358 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
20:49:26.029124 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'

20:49:26.033910 trace git-lfs: attempting pure SSH protocol connection

20:49:26.035187 trace git-lfs: run_command: ssh [email protected] git-lfs-transfer haikusw/isowords.git upload
20:49:26.036802 trace git-lfs: exec: ssh '[email protected]' 'git-lfs-transfer haikusw/isowords.git upload'

20:49:27.039835 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF

20:49:27.041305 trace git-lfs: Upload refs [async] to remote haikusw_origin
20:49:27.041744 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
20:49:27.052994 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
20:49:27.060126 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'haikusw_origin'

20:49:28.276084 trace git-lfs: attempting pure SSH protocol connection

20:49:28.277091 trace git-lfs: run_command: ssh [email protected] git-lfs-transfer haikusw/isowords.git upload
20:49:28.277299 trace git-lfs: exec: ssh '[email protected]' 'git-lfs-transfer haikusw/isowords.git upload'

20:49:29.244726 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF

20:49:29.245339 trace git-lfs: run_command: ssh [email protected] git-lfs-authenticate haikusw/isowords.git upload
20:49:29.245621 trace git-lfs: exec: ssh '[email protected]' 'git-lfs-authenticate haikusw/isowords.git upload'
20:49:30.318900 trace git-lfs: exec: /usr/bin/security 'list-keychains'
20:49:30.357009 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'lfs.github.com' '/Library/Keychains/System.keychain'
20:49:30.374200 trace git-lfs: HTTP: POST https://lfs.github.com/haikusw/isowords/locks/verify
> POST /haikusw/isowords/locks/verify HTTP/1.1
> Host: lfs.github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: RemoteAuth gitauth-v1-ga2G9UNrcUDLsS2bddhFKAHLLUjpHCo6sBfURxbQmhRl9lZElZLOYnzYKoKmbWVtYmVys3VzZXI6MjIyMjcxOmhhaWt1c3elcHJvdG-jc3No
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}

20:49:30.851353 trace git-lfs: HTTP: 200

< HTTP/2.0 200 OK
< Content-Length: 41
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 03:49:30 GMT
< X-Github-Request-Id: ECF7:419E:2A5BD8:7E3DC4:627C83CA
< 
20:49:30.851456 trace git-lfs: HTTP: {"ours":[],"theirs":[],"next_cursor":""}

{"ours":[],"theirs":[],"next_cursor":""}
20:49:30.853497 trace git-lfs: tq: running as batched queue, batch size of 100
20:49:30.853932 trace git-lfs: run_command: git rev-list --objects --ignore-missing --not --remotes=haikusw_origin --stdin --
20:49:30.856315 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
20:49:30.857915 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
push 73b2f912875308b6669232c2bad2456ae93f5aae954bc44cc2e45a0a50d34fdc => App/AppClip/Assets.xcassets/AppIcon.appiconset/[email protected]

... many lines like the above for each asset ...

20:49:30.921261 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
20:49:30.921284 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
20:49:30.921312 trace git-lfs: filepathfilter: accepting "tmp"

It's odd that SSH connection isn't working. I tried connecting to GitHub via SSH again via ssh -T [email protected] and turned on verbose and it noted:

debug1: Authentication succeeded (publickey).
Authenticated to github.com ([140.82.114.4]:22).

and is using my ed25519 type 3 ssh key.

(there's a lot of debug output from ssh -Tv and I'm not sure how much of that is kosher to post).

Well, hopefully some of that leads to an "ah ha!" moment and either clarity on a bug or something I'm doing wrong or ?

haikusw avatar May 12 '22 04:05 haikusw

The other thing I thought of later:

Why does having the SSH= as part of the fork root impact the connection to the forked repo?

Endpoint=https://github.com/pointfreeco/isowords.git/info/lfs (auth=none)
  [email protected]:pointfreeco/isowords.git
Endpoint (haikusw_origin)=https://[email protected]/haikusw/isowords.git/info/lfs (auth=none)

The SSH=git… is for the pointfreeco repo which I don't have write access too, and shouldn't need write access to.

The haikusw_origin Endpoint is just a pure HTTPS entry with no SSH aspect to it. So why does an aspect on the fork root have anything to do with the connection to verify locks on the dest repo?

The second question is: why is it failing when ssh -T [email protected] succeeds?

Realized I hadn't posted the full output of the failing connection, so let me post the here.

*** Odd thing is, I don't see it doing any authentication at all for this HTTP POST request.. and that just seems like a bug.

Perhaps I'm missing something.

% GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async
08:32:10.571461 git.c:745               trace: exec: git-lfs push --dry-run haikusw_origin async
08:32:10.571963 run-command.c:654       trace: run_command: git-lfs push --dry-run haikusw_origin async
08:32:10.646061 trace git-lfs: exec: git 'version'
08:32:10.651554 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
08:32:10.657385 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
08:32:10.664772 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
08:32:10.667904 trace git-lfs: exec: git 'config' '--includes' '-l'
08:32:10.670946 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
08:32:10.673662 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
08:32:10.677387 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
08:32:10.682163 trace git-lfs: Upload refs [async] to remote haikusw_origin
08:32:10.682243 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
08:32:10.685946 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
08:32:10.688650 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'haikusw_origin'
08:32:11.166265 trace git-lfs: exec: /usr/bin/security 'list-keychains'
08:32:11.190638 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'github.com' '/Library/Keychains/System.keychain'
08:32:11.207274 trace git-lfs: HTTP: POST https://[email protected]/haikusw/isowords.git/info/lfs/locks/verify
> POST /haikusw/isowords.git/info/lfs/locks/verify HTTP/1.1
> Host: github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}08:32:11.652653 trace git-lfs: HTTP: 403


< HTTP/2.0 403 Forbidden
< Content-Length: 154
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 15:32:11 GMT
< X-Frame-Options: DENY
< X-Github-Request-Id: C59B:59A6:B2F112:18C7D70:627D287B
< 
08:32:11.652738 trace git-lfs: HTTP: {"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"C59B:59A6:B2F112:18C7D70:627D287B"}

{"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"C59B:59A6:B2F112:18C7D70:627D287B"}
error: Authentication error: Authentication required: You must have push access to verify locks

Just to be 100% sure this is the issue in my original repo that shows this issue I removed the origin repo and added it back as an HTTPS origin:

% git remote remove origin
% git remote add origin https://github.com/pointfreeco/isowords.git
% git remote -v
haikusw_origin	https://[email protected]/haikusw/isowords.git (fetch)
haikusw_origin	https://[email protected]/haikusw/isowords.git (push)
origin	https://github.com/pointfreeco/isowords.git (fetch)
origin	https://github.com/pointfreeco/isowords.git (push)

I then re-did the push command again and it fails in the same way even though the remote doesn't have the SSH bit on it for the root repo.

So, perhaps not the actual issue...?

Happy to try anything else you think might be useful / illuminating.

Here's the failure after changing origin to an HTTPS one in that working copy (still isn't visibly doing any authentication on that HTTPS POST call that I see in the logging?):

% GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async
08:43:07.022683 git.c:745               trace: exec: git-lfs push --dry-run haikusw_origin async
08:43:07.023298 run-command.c:654       trace: run_command: git-lfs push --dry-run haikusw_origin async
08:43:07.063859 trace git-lfs: exec: git 'version'
08:43:07.068549 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
08:43:07.072833 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
08:43:07.077491 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
08:43:07.080421 trace git-lfs: exec: git 'config' '--includes' '-l'
08:43:07.083359 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
08:43:07.085969 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
08:43:07.089603 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
08:43:07.094018 trace git-lfs: Upload refs [async] to remote haikusw_origin
08:43:07.094082 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
08:43:07.097252 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
08:43:07.099927 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'haikusw_origin'
08:43:07.556945 trace git-lfs: exec: /usr/bin/security 'list-keychains'
08:43:07.588748 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'github.com' '/Library/Keychains/System.keychain'
08:43:07.608552 trace git-lfs: HTTP: POST https://[email protected]/haikusw/isowords.git/info/lfs/locks/verify
> POST /haikusw/isowords.git/info/lfs/locks/verify HTTP/1.1
> Host: github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}08:43:08.074734 trace git-lfs: HTTP: 403


< HTTP/2.0 403 Forbidden
< Content-Length: 153
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 15:43:08 GMT
< X-Frame-Options: DENY
< X-Github-Request-Id: C5F0:67FD:181BB4:252D10:627D2B0B
< 
08:43:08.074941 trace git-lfs: HTTP: {"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"C5F0:67FD:181BB4:252D10:627D2B0B"}

{"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"C5F0:67FD:181BB4:252D10:627D2B0B"}
error: Authentication error: Authentication required: You must have push access to verify locks

haikusw avatar May 12 '22 15:05 haikusw

Thanks for the all the detail! Just to summarize where I believe things stand, and make a few notes:

  1. It sounds like you have a workaround by using an HTTPS endpoint, which is great. Please correct me if I'm wrong about that.
  2. You asked about the where the SSH part comes from here in the git lfs env output: 
    Endpoint=https://github.com/pointfreeco/isowords.git/info/lfs (auth=none)
  [email protected]:pointfreeco/isowords.git
    That means that Git LFS was configured to use either a ssh:// URL or a "bare" [email protected]:... connection string, not an https:// one. (The HTTPS line is a bit misleading because when an SSH URL is provided the HTTPS one shown by git lfs env is constructed by Git LFS as a "backup" URL from the configured SSH one; it's not actually from your configuration.) You might look for that in a .lfsconfig or .git/config file in your original local repo (or in your global ~/.gitconfig, but that seems unlikely as it would affect your other recent fresh test clones as well). It would look something like (maybe with a ssh:// prefix): 
    [lfs]
	url = [email protected]:pointfreeco/isowords.git
  3. You asked why the SSH connection isn't working re the example where the HTTPS connection works (in this comment above). I suspect this is because you noticed the pure SSH protocol connection failed error message in the trace output. If so, that's a red herring, and one I've been caught by too. Git LFS recently introduced a "pure SSH" transfer protocol on the client side, but no one has implemented it in production on the server side yet. So the client checks for support and then falls back to using HTTPS for the actual transfer; it only uses SSH for the authentication and authorization. You can see that in that example the SSH authorization did succeed; there are no error messages after the exec: ssh '[email protected]' 'git-lfs-authenticate haikusw/isowords.git upload' trace line.

What I'm interested in is how to reproduce the failing case. I suspect this might have something to do with an [lfs] configuration in that repo's .lfsconfig or .git/config files, which would tell Git LFS to use an SSH (or "bare") URL for its endpoint. Note that this is different from the remotes you have configured with normal Git via git remote.

I'm still unsure if this is a client-side bug or a GitHub-side one.

chrisd8088 avatar May 12 '22 18:05 chrisd8088

One thing that might help a bit is if you could post the output of git config -l in your original repo where things are failing. Just make sure to redact any personal info like authorization credentials or email addresses.

chrisd8088 avatar May 12 '22 18:05 chrisd8088

Thanks for the all the detail! Just to summarize where I believe things stand, and make a few notes:

  1. It sounds like you have a workaround by using an HTTPS endpoint, which is great. Please correct me if I'm wrong about that.

I have a second clone using and HTTPS endpoint and it works. The original cloned repo still doesn't work. Simply changing the remote to an HTTPS url in that clone didn't fix it (the last section of my most recent post details this).

So I'm still curious why that one continues to fail (and interested in helping track it down, if we can).

  1. You asked about the where the SSH part comes from here in the git lfs env output: …

Very interesting!

I looked and I'm not seeing anything lfs related that overrides the endpoint in ~/.gitconfig, or the problematic clone's .git/config and no .lfsconfig files locally or globally that I'm seeing.

Here's the output of git config -l from the directory that doesn't work (after having done the remote change to an HTTPS endpoint as noted above), as you requested:

% git config -l
credential.helper=osxkeychain
mergetool.keepbackup=true
user.name=Tyler
user.email={REDACTED}
core.excludesfile=/Users/tyler/.gitignore_global
difftool.sourcetree.cmd=opendiff "$LOCAL" "$REMOTE"
difftool.sourcetree.path=
mergetool.sourcetree.cmd=/Applications/Sourcetree.app/Contents/Resources/opendiff-w.sh "$LOCAL" "$REMOTE" -ancestor "$BASE" -merge "$MERGED"
mergetool.sourcetree.trustexitcode=true
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
difftool.Kaleidoscope.cmd=ksdiff --partial-changeset --relative-path "$MERGED" -- "$LOCAL" "$REMOTE"
mergetool.Kaleidoscope.cmd=ksdiff --merge --output "$MERGED" --base "$BASE" -- "$LOCAL" --snapshot "$REMOTE" --snapshot
mergetool.Kaleidoscope.trustexitcode=true
commit.template=/Users/tyler/.stCommitMsg
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.haikusw_origin.url=https://[email protected]/haikusw/isowords.git
remote.haikusw_origin.gtserviceaccountidentifier= {REDACTED}
remote.haikusw_origin.fetch=+refs/heads/*:refs/remotes/haikusw_origin/*
branch.haikusw_main.remote=haikusw_origin
branch.haikusw_main.merge=refs/heads/main
user.name=haikuty
user.email= {REDACTED}
lfs.repositoryformatversion=0
remote.origin.url=https://github.com/pointfreeco/isowords.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
  1. … Thanks for the explanation. Makes sense now :)

What I'm interested in is how to reproduce the failing case. I suspect this might have something to do with an [lfs] configuration in that repo's .lfsconfig or .git/config files, which would tell Git LFS to use an SSH (or "bare") URL for its endpoint. Note that this is different from the remotes you have configured with normal Git via git remote.

Me also! I have included (above) the git config -l you requested and I looked for any .lfsconfig files (none) or related entries in git config files (not seeing any).
Let me know if there's something else I can do to help.

I'm still unsure if this is a client-side bug or a GitHub-side one.

:) Or a user error (always a possibility!). I appreciate your interest in tracking this down!

From GitHub support:

The engineering team are keeping an eye on the public issue you created so they can see everything going on there. If there's anything they need you to do in addition, or any output they need that shouldn't be posted publicly, I'll be back in touch!

So if we can figure out root cause and it's on GitHub server side, they should hear about it.

(I'm going to be really embarrassed if it's something dumb I've done :), but this has come up enough times for others here that even if that's the issue we can at least get it documented - "don't do this dumb thing" - to help others :)). Thanks again for your assistance in tracking this down.

haikusw avatar May 12 '22 19:05 haikusw

Ok. New Test. Used git tower to checkout the root repo and then added my fork as an additional remote and then checked out the async branch again and then tried the push of that branch to my remote and it failed the same way.

So there's something about the clone that Tower produces that is triggering this.

Just to confirm, I made a clone of the root repo on the command line, checked out the async branch, added my remote repo, and then tried the git lfs push of the async branch to my repo and the dry run succeeded.

I then compared the output of git config -l for the command line clone that works and the tower created one that doesn't.

The key line seems likely to be this one:

lfs.https://github.com/haikusw/isowords.git/info/lfs.access=basic

which is in the command line one but not in the Tower created one.

haikusw avatar May 12 '22 19:05 haikusw

Hmm. Now somehow I made a command line clone that doesn't have that line and it is failing. So that line in the git config -l seems to be the difference.

haikusw avatar May 12 '22 20:05 haikusw

Ah, more complicated than that.

Either the destination remote needs to not have the user name in the url:

https://[email protected]/haikusw/isowords.git // fails unless… (see below)

vs

https://github.com/haikusw/isowords.git // always works

What makes this complicated to test back and forth in a single repo is that once you have successfully done a git lfs command in a repo it seems to add:

[lfs "https://github.com/haikusw/isowords.git/info/lfs"]
	access = basic

to .git/config for that repo.

Once git-lfs has done that, then the username in the remote url no longer causes authentication to fail.

So, the authentication issue is reproducible when there is a username in the destination remote url AND lfs.https access is not overridden to "basic" via the config file.

Thus, to reproduce this you can do the following:

  1. on github.com: Fork the https://github.com/pointfreeco/isowords to your GitHub account so you have a fork with write access (let's say it's at https://github.com/yourname/isowords )

  2. in terminal:

git clone https://github.com/pointfreeco/isowords.git
git checkout -b test42
git remote add my_origin https://[email protected]/yourname/isowords.git
git lfs push --dry-run my_origin test42

and that last command should fail with the authentication error. (Though likely this will depend on something else relating to ssh keys, personal access tokens, etc, but we'll have to start by seeing if you can reproduce it with your current setup and if not we can figure out what is different between our setups).

...

  1. To then make the git lfs push work, either: 3a) add the following to your .git/config file manually:

    [lfs "https://github.com/yourname/isowords.git/info/lfs"]
	access = basic`

    3b) or, change your remote to no longer include your username: git remote set-url my_origin https://github.com/yourusername/isowords.git

Note that if you do (3b) and then do the git lfs push command (or any other git lfs command?) it will add the entry from (3a) to your .git/config file. Once that's done you can switch the git remote url back to including your username and it will then work.

Anyway, that's how to reproduce the issue there. (let me know if you have any trouble reproducing it).

Now the question is: Why does having non-basic authentication with a remote with a user name in the url cause authentication to fail.

As a reminder of things that may be involved:

  • I have a .ssh key added to my GitHub account.
  • I have a Personal Access Token set for Tower and echo "host=github.com\nprotocol=https" | git credential fill seems to have it set

Not sure what else.

haikusw avatar May 12 '22 21:05 haikusw

Hey, thanks for all the investigation! I'll try to use your steps to reproduce it myself in the next few days, and with luck that will shed some light on the source of the problem. Thanks again!

chrisd8088 avatar May 13 '22 05:05 chrisd8088

Welcome. Thanks for taking a look. I look forward to hearing what you discover.

haikusw avatar May 13 '22 17:05 haikusw

The reproduction steps were extremely helpful, thank you! I'm not entirely sure yet what the best way to resolve the problem is, but at least I think I mostly understand where it's coming from.

We can close out as a possibility is any issue with GitHub's API. They are responding appropriately to what the client sends.

One tricky thing about this issue is that it appears to be the result of some logic in the Go HTTP client, which sets an Authorization: Basic <base64-creds> header when the URL passed to the client has a username in it, even if it lacks any password. The Git LFS server (in your case, GitHub's API) sees this, determines that the encoded credentials are invalid, and correctly returns a 403. That in turn halts the Git LFS client from proceeding further (again, correctly).

Even more tricky is the fact that this added Authorization header is not output by the DumpRequest() or DumpRequestOut() functions, which are what Git LFS relies on to generate its trace output. So that's why you don't see them in your various tracing attempts above.

But the following small Go program demonstrates this behaviour of the Go libraries if one compiles and runs it with the GODEBUG=http2debug=1 environment variable. You can also set GODEBUG=http2debug=1 and run the failing Git LFS command to see how the extra header is injected by Go.

package main

import (
        "fmt"
        "net/http"
        "net/http/httputil"
)

func main() {
        req, _ := http.NewRequest("GET", "https://[email protected]", nil)
        dump, _ := httputil.DumpRequestOut(req, false)
        fmt.Printf("%s\n", dump)
        c := &http.Client{}
        c.Do(req)
}

$ go build demo.go
$ GODEBUG=http2debug=1 ./demo

GET / HTTP/1.1
Host: play.golang.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

2022/05/15 22:52:54 http2: Transport failed to get client conn for play.golang.com:443: http2: no cached connection was available
2022/05/15 22:52:54 http2: Transport creating client conn 0xc0000aa180 to [2607:f8b0:400a:805::2011]:443
2022/05/15 22:52:54 http2: Transport encoding header ":authority" = "play.golang.com"
2022/05/15 22:52:54 http2: Transport encoding header ":method" = "GET"
2022/05/15 22:52:54 http2: Transport encoding header ":path" = "/"
2022/05/15 22:52:54 http2: Transport encoding header ":scheme" = "https"
2022/05/15 22:52:54 http2: Transport encoding header "authorization" = "Basic Zm9vOg=="
2022/05/15 22:52:54 http2: Transport encoding header "accept-encoding" = "gzip"
2022/05/15 22:52:54 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2022/05/15 22:52:54 http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=65536
2022/05/15 22:52:54 http2: Transport received WINDOW_UPDATE len=4 (conn) incr=983041
2022/05/15 22:52:54 http2: Transport received SETTINGS flags=ACK len=0
2022/05/15 22:52:54 http2: Transport received HEADERS flags=END_HEADERS stream=1 len=233

Notice the authorization header line with the value Basic Zm9vOg==" (which is foo:, base64-encoded).

In terms of a resolution to this, I think we probably want to strip out the user credentials from the URL we pass to the Go HTTP client, in order to avoid triggering this behaviour, and then only pass them via an explicit Authorization header if both user and password are set. But there are a few things to consider here, like whether we want to discourage people from putting passwords into configuration files in the first place, and also where we need to make the changes so as to cover all our HTTP client use cases. And whatever changes we make should have some tests, of course.

At any rate, thank you again for your patience and effort creating a reproduction case! Having one made it much easier to eventually track down the cause.

chrisd8088 avatar May 16 '22 06:05 chrisd8088

Very cool!

Sounds like you had a fun bug hunt!

The observed behavior totally makes sense based on what you discovered. Nice wrinkle on the DumpRequest output making debugging more challenging :)

30+ years of software development means I know just how useful it is to have a reproducible case (!). Thanks for taking advantage of it and spending the time to work through the steps and find root cause!

While wanting to discourage password use in configuration files makes sense, I'd consider also the use of GUI apps that use git-lfs but use a reasonably secure method like operating system keychains to store passwords.

I'm under the impression that Github's Personal Access Tokens for use with such apps are passed using the Authentication header, but I haven't actually verified that. If so it would be unfortunate to make that no longer work.

Thanks again for doing the work to track this down. Given how often I saw folks stumbling on this over the years it'll be great to get it fixed.

haikusw avatar May 16 '22 06:05 haikusw

I think I see the same problem. Are there some workarounds other than disabling git-lfs?

AlekSi avatar Oct 07 '22 10:10 AlekSi

@AlekSi

I think I see the same problem. Are there some workarounds other than disabling git-lfs?

The details here are where I narrowed it down to having the user name in the git repo URL and how changing that makes it always work. That's written up more for reproducing the issue and less as a workaround, but see if reading that lets you figure out the workaround. If not I can try to rewrite that.

haikusw avatar Oct 07 '22 14:10 haikusw

@haikusw , This is great.

Basically what's needed in the config file:

[lfs]
    access = basic

RoyiAvital avatar Jan 18 '23 12:01 RoyiAvital

That simple workaround does not work for me, unfortunately

AlekSi avatar Feb 01 '23 19:02 AlekSi

I know the issue is a bit old but wanted to add something I just ran into, not sure if strictly related to the go client, or an issue in what git lfs is passing to it.

When I run GODEBUG=http2debug=1 git push (in a scenario like described above in other comments; the upstream repo has lfs enabled, I have a fork of it, but in my case I have write access to both; and I'm pushing the branch to my fork), the Authorization header I see in the trace output is missing a few characters in the base64 encoded part. When I base64-decode the value from the header, I get PersonalAccessToken:<redacted>zu88%, with a % at the end that shouldn't be there. If I base64 encode the correct string without the percent symbol, I get a different string from the one being sent in the authorization header. The string in the header is exactly 80 characters long (missing 4 chars), in case that points to something in particular.

This is with

> git lfs --version
git-lfs/3.4.0 (GitHub; linux amd64; go 1.20.6; git d06d6e9e)

That said, even if I craft a request manually to the /locks/verify endpoint with the correct header, I'm still getting a "You must have push access to verify locks" error 🤔 so I'm probably also running into a separate problem.

alexvy86 avatar Aug 25 '23 16:08 alexvy86