Add pure SSH LFS support

[ConcurrentCrab]

trafficstars

Fixes #17554 /claim #17554

To test, run pushes like: GIT_TRACE=1 git push. The trace output should mention "pure SSH connection".

[ConcurrentCrab]

@lunny added copyright headers.

[ConcurrentCrab]

Improved error handling.

[ConcurrentCrab]

Accidentally resolved the conversation about DB connections above, but reiterating:

Pushes and clones work... but there's sometimes (by which I mean decently reproducible) bug where after transferring and verification and everything... the connection just hangs for almost exactly 120 seconds. After which it just ends gracefully and git proceeds with its push.

It's particularly nasty, I have no leads on it yet. The best I can figure out is that it looks like the git-lfs client seems to not send the quit command that it should at the end of the protocol, or that it isn't coming through.

[ConcurrentCrab]

Added locking API support.

Well at least I figured out what's causing the hang. Setting lfs.ssh.automultiplex to false on the client side: https://github.com/git-lfs/git-lfs/blob/8e36a03d85bf05bbca004dd7e8b55b147809b3e0/docs/man/git-lfs-config.adoc?plain=1#L90 gets rid of the hang. The bad news is that it's true by default.

I think this is feature interacting badly with the SSH server?

I traced the traffic, and indeed, something weird is happening on the client side. Git LFS spawns several workers each making its own connection. In a multiplexed connection, only the "first" spawn of the our commands gets a quit message. The rest of them have to wait 120 seconds for an EOF. I'm almost tempted to believe this is a bug on the git-lfs client side.

Can you check and see if this is reproducible in your environments?

[ConcurrentCrab]

Yeah, I'm not going crazy. This commit: https://github.com/git-lfs/git-lfs/commit/44b8801cbb57891b4c66fc3c43551802e5532bd1 breaks pure SSH LFS sessions.

And this has history, it isn't even the first time lol: https://github.com/git-lfs/git-lfs/pull/5537

[ConcurrentCrab]

just made: https://github.com/git-lfs/git-lfs/pull/5816

[ConcurrentCrab]

@lafriks adressed the url joins.

[ConcurrentCrab]

@techknowlogick dropped the vendoring commit.

I believe everything looks good from my side now.

If this PR looks satisfactory to you, and once you've tested and verified that all functionality works, the matter of https://github.com/git-lfs/git-lfs/pull/5816 still remains. I have a feeling it's going to take a while, but even after a release with that fix, you'd probably still want to wait for some time to let the new version become common enough that users don't get the deadlock.

So how would you like to proceed with this? One option I can thank of is: I can split this PR into 2, we can merge the refactor-y commits now (to minimise chances of merge conflicts later), and the commits that actually implement this functionality remain in this PR, which you can merge whenever you deem appropriate.

[ConcurrentCrab]

Also, there's no way to selectively identify older git-lfs clients and just disable the SSH protocol for them, as the protocol doesn't have a user-agent identifier (it's been discussed several times as a deficiency in the protocol, but no improvements have been implemented).

[ConcurrentCrab]

Alternatively, I suppose, we can add a default-false config option to let users decide when it would be appropriate for their needs? It can be turned default-true once the "updated LFS client" utopia has been achieved.

[ConcurrentCrab]

Hey all,

Just wanted to do a quick ping to ask if you've had the chance to look at this yet.

[ConcurrentCrab]

@techknowlogick added a new commit adding a LFS_ALLOW_PURE_SSH option to the server section. It's default-false due to the zero value. It can be later set to default-true. But like I said above, it needs to be after a point in time where most people have shifted to the fixed LFS build (it'll probably be a while considering almost-LTS environments like ubuntu).

Also rebased on main.

[ConcurrentCrab]

The documentation blurb could be something like:

LFS Pure SSH Protocol Support The LFS protocol supports connecting purely over SSH (without exposing an HTTP endpoint). Support for it can be enabled with the config option server.LFS_ALLOW_PURE_SSH.

NOTE: The option is currently set to default false due to an open bug in the git-lfs client that causes transfers to hang: https://github.com/git-lfs/git-lfs/pull/5816 This can be worked around on the client machines by setting the git config: git config --global lfs.ssh.automultiplex false

[lunny]

The documentation blurb could be something like:

LFS Pure SSH Protocol Support The LFS protocol supports connecting purely over SSH (without exposing an HTTP endpoint). Support for it can be enabled with the config option server.LFS_ALLOW_PURE_SSH. NOTE: The option is currently set to default false due to an open bug in the git-lfs client that causes transfers to hang: git-lfs/git-lfs#5816 This can be worked around on the client machines by setting the git config: git config --global lfs.ssh.automultiplex true

Please send a doc pull request to https://gitea.com/gitea/docs which should be merged after this one done.

[ConcurrentCrab]

The documentation blurb could be something like:

LFS Pure SSH Protocol Support The LFS protocol supports connecting purely over SSH (without exposing an HTTP endpoint). Support for it can be enabled with the config option server.LFS_ALLOW_PURE_SSH. NOTE: The option is currently set to default false due to an open bug in the git-lfs client that causes transfers to hang: git-lfs/git-lfs#5816 This can be worked around on the client machines by setting the git config: git config --global lfs.ssh.automultiplex true

Please send a doc pull request to https://gitea.com/gitea/docs which should be merged after this one done.

@lunny where should this be in the docs?

[lunny]

The documentation blurb could be something like:

LFS Pure SSH Protocol Support The LFS protocol supports connecting purely over SSH (without exposing an HTTP endpoint). Support for it can be enabled with the config option server.LFS_ALLOW_PURE_SSH. NOTE: The option is currently set to default false due to an open bug in the git-lfs client that causes transfers to hang: git-lfs/git-lfs#5816 This can be worked around on the client machines by setting the git config: git config --global lfs.ssh.automultiplex true

Please send a doc pull request to https://gitea.com/gitea/docs which should be merged after this one done.

@lunny where should this be in the docs?

I think at least https://gitea.com/gitea/docs/src/branch/main/docs/administration/config-cheat-sheet.md should be updated and also https://gitea.com/gitea/docs/src/branch/main/docs/administration/git-lfs-support.md

[ConcurrentCrab]

@lunny made https://gitea.com/gitea/docs/pulls/49

[lunny]

Linter and check failure should be fixed.

[ConcurrentCrab]

@delvh changed to use container.Set.

[ConcurrentCrab]

Whoops, forgot to update some comments.

[ConcurrentCrab]

Hey all,

so I believe that the current reviews (all those that had consensus, anyway) have been addressed.

[6543]

buildin ssh server + sha256 repo does not work ...

[6543]

sha1 also does not work:

GIT_TRACE=1 git push --verbose                                                                                                                    on  main [⇡]
21:56:42.989596 git.c:463               trace: built-in: git push --verbose
Pushing to ssh://192.168.1.4:20435/test/frwwa.git
21:56:42.989925 run-command.c:657       trace: run_command: unset GIT_PREFIX; ssh -p 20435 [email protected] 'git-receive-pack '\''/test/frwwa.git'\'''
21:56:43.160171 run-command.c:657       trace: run_command: .git/hooks/pre-push origin ssh://[email protected]:20435/test/frwwa.git
21:56:43.165891 git.c:750               trace: exec: git-lfs pre-push origin ssh://[email protected]:20435/test/frwwa.git
21:56:43.165941 run-command.c:657       trace: run_command: git-lfs pre-push origin ssh://[email protected]:20435/test/frwwa.git
21:56:43.174518 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote' '-v'
21:56:43.175729 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
21:56:43.176843 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
21:56:43.177865 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
21:56:43.179094 trace git-lfs: exec: git 'config' '--includes' '-l'
21:56:43.179921 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
21:56:43.180770 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
21:56:43.181805 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
21:56:43.183147 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
21:56:43.184646 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
21:56:43.186191 trace git-lfs: attempting pure SSH protocol connection
21:56:43.186199 trace git-lfs: spawning pure SSH connection
21:56:43.186244 trace git-lfs: run_command: ssh -oControlMaster=yes -oControlPath=/run/user/1000/sock-73099538/lfs.sock -p 20435 [email protected] git-lfs-transfer /test/frwwa.git upload
21:56:43.186426 trace git-lfs: exec: ssh '-oControlMaster=yes' '-oControlPath=/run/user/1000/sock-73099538/lfs.sock' '-p' '20435' '[email protected]' 'git-lfs-transfer /test/frwwa.git upload'
21:56:43.349324 trace git-lfs: pure SSH connection successful
21:56:43.349337 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): unexpected EOF
21:56:43.349537 trace git-lfs: pre-push: refs/heads/main 954452e504825b1302f181bb9930fcc06e3632be refs/heads/main e093361042867374edf0f1112e14a5bdc395d31c
21:56:43.349639 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
21:56:43.352266 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'origin'
21:56:43.542290 trace git-lfs: attempting pure SSH protocol connection
21:56:43.542302 trace git-lfs: spawning pure SSH connection
21:56:43.542331 trace git-lfs: run_command: ssh -oControlMaster=yes -oControlPath=/run/user/1000/sock-290582623/lfs.sock -p 20435 [email protected] git-lfs-transfer /test/frwwa.git upload
21:56:43.542506 trace git-lfs: exec: ssh '-oControlMaster=yes' '-oControlPath=/run/user/1000/sock-290582623/lfs.sock' '-p' '20435' '[email protected]' 'git-lfs-transfer /test/frwwa.git upload'
21:56:43.711956 trace git-lfs: pure SSH connection successful
21:56:43.711968 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): unexpected EOF
21:56:43.712109 trace git-lfs: run_command: ssh -p 20435 [email protected] git-lfs-authenticate /test/frwwa.git upload
21:56:43.712273 trace git-lfs: exec: ssh '-p' '20435' '[email protected]' 'git-lfs-authenticate /test/frwwa.git upload'
21:56:43.881384 trace git-lfs: HTTP: POST http://192.168.1.4:3000/test/frwwa.git/info/lfs/locks/verify
21:56:43.882774 trace git-lfs: HTTP: 200
21:56:43.882936 trace git-lfs: HTTP: {"ours":[],"theirs":[]}

Locking support detected on remote "origin". Consider enabling it with:
  $ git config lfs.https://192.168.1.4/test/frwwa.git/info/lfs.locksverify true
21:56:43.883728 trace git-lfs: tq: running as batched queue, batch size of 100
21:56:43.884139 trace git-lfs: run_command: git rev-list --objects --ignore-missing --stdin --
21:56:43.884182 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-list' '--objects' '--ignore-missing' '--stdin' '--'
21:56:43.884913 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
21:56:43.885129 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
21:56:43.887182 trace git-lfs: tq: sending batch of size 1                                                                                                                            
21:56:43.887311 trace git-lfs: ssh cache: [email protected] git-lfs-authenticate /test/frwwa.git upload
21:56:43.887424 trace git-lfs: api: batch 1 files
21:56:43.887498 trace git-lfs: HTTP: POST http://192.168.1.4:3000/test/frwwa.git/info/lfs/objects/batch
21:56:43.889318 trace git-lfs: HTTP: 200
21:56:43.889357 trace git-lfs: HTTP: {"objects":[{"oid":"99405f7b89bfa84f1200633b42c0ad3a12694e884e091eb7f3bb35d31d69cf2f","size":9,"actions":{"upload":{"href":"http://192.168.1.4:3000/test/frwwa.git/info/lfs/objects/99405f7b89bfa84f1200633b42c0ad3a12694e884e091eb7f3bb35d31d69cf2f/9","header":{"Authorization":"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJSZXBvSUQiOjksIk9wIjoidXBsb2FkIiwiVXNlcklEIjoxLCJleHAiOjE3MjQzNTY2MDMsIm5iZiI6MTcyNDI3MDIwM30.IHJSUGwz3iXVmjQCd-2WlpwhqDi5mbZwmmJC3c_Zh7I"}},"verify":{"href":"http://192.168.1.4:3000/te
21:56:43.889385 trace git-lfs: HTTP: st/frwwa.git/info/lfs/verify","header":{"Accept":"application/vnd.git-lfs+json;q=0.9, */*;q=0.8","Authorization":"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJSZXBvSUQiOjksIk9wIjoidXBsb2FkIiwiVXNlcklEIjoxLCJleHAiOjE3MjQzNTY2MDMsIm5iZiI6MTcyNDI3MDIwM30.IHJSUGwz3iXVmjQCd-2WlpwhqDi5mbZwmmJC3c_Zh7I"}}}}]}

21:56:43.889453 trace git-lfs: tq: starting transfer adapter "basic"
21:56:43.889726 trace git-lfs: HTTP: PUT http://192.168.1.4:3000/test/frwwa.git/info/lfs/objects/99405f7b89bfa84f1200633b42c0ad3a12694e884e091eb7f3bb35d31d69cf2f/9
21:56:43.907354 trace git-lfs: HTTP: 200
21:56:43.907401 trace git-lfs: HTTP: {"Message":"OK"}

21:56:43.907481 trace git-lfs: tq: verify 99405f7 attempt #1 (max: 3)
21:56:43.907638 trace git-lfs: HTTP: POST http://192.168.1.4:3000/test/frwwa.git/info/lfs/verify
21:56:43.908799 trace git-lfs: HTTP: 200
21:56:43.908958 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore                                                                                              
21:56:43.908967 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
21:56:43.908996 trace git-lfs: filepathfilter: accepting "tmp"
21:56:43.910889 run-command.c:657       trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
21:56:43.912172 git.c:463               trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 16 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 65.45 MiB | 48.88 MiB/s, done.
Total 5 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: . Processing 1 references
remote: Processed 1 references in total
To ssh://192.168.1.4:20435/test/frwwa.git
   e093361..954452e  main -> main
updating local tracking ref 'refs/remotes/origin/main'

[techknowlogick]

@6543 what OS, and version of git are you using?

[ConcurrentCrab]

buildin ssh server + sha256 repo does not work

sha1 also does not work:

21:56:43.711956 trace git-lfs: pure SSH connection successful
21:56:43.711968 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): unexpected EOF

@6543 did you enable the config flag? That error message is what it says when SSH LFS is disabled.

[6543]

@6543 what OS, and version of git are you using?

NixOS | Linux 6.10.1-zen1 x86_64

git version 2.44.1 git-lfs/3.5.1 (3.5.1; linux amd64; go 1.22.5)

do i have to enable it explizite via some config? as i understand it should transparently work

[ConcurrentCrab]

@6543 what OS, and version of git are you using?

NixOS | Linux 6.10.1-zen1 x86_64

git version 2.44.1 git-lfs/3.5.1 (3.5.1; linux amd64; go 1.22.5)

do i have to enable it explizite via some config? as i understand it should transparently work

@6543 That would indeed have been ideal, but see: https://github.com/git-lfs/git-lfs/pull/5816 (also https://gitea.com/gitea/docs/pulls/49)

[SherryYanmengTang]

May I know when we could get this PR merged?

[lunny]

The LFS requests MUST be from an internal API but not an external API, otherwise, it's the same as HTTP based LFS support or worse because there is a SSH proxy.

[6543]

I can confirm with https://github.com/git-lfs/git-lfs/pull/5816 it works ... we should add this to the docs untill it's fixed!