random 401 status in lfs.log causing client-side smudge errors

[KC-FLIR]

We use the lfs-test server to host the binary files in one of our git repos. This is on-premises and it happily runs on a Centos server.

Except that it randomly will throw a 401 status when asked for file. The git repo has about 100 binary files in the tree, and there's no telling when you'll get the 401. If you repeatedly pull, the client-side cache will eventually populate and the "error goes away" for the user.

There seems to be some sort of caching issue, because once ONE developer has finally got their repo synced for a particular commit, the next developer that pulls that commit from the git repo has NO trouble getting the latest 97 files from the lfs repo. The server seems to have no problems finding the files and supplying them.

If a dev pushes a new set of files, they will cause this grief once again when everyone pulls their new work - right up to the point the server has magically cached all of them and can now start serving them all without a problem.

We use https and authentication through the management interface with a user set up. Of course, git lfs now complains at every point that this isn't good enough, even if it did work fine when we first started with lfs all those months ago.

I have attached the server log and a client error log.

Is there something I'm missing? We run 9 instances of the test-lfs server for different repos (all instances have their own ports assigned) - and most seem absolutely fine.. but this one it a right pain. It does have the biggest number of binary files per commit though.

20171123T114847.555849927.log

lfs.log

the lfs server is 0.3.0, my client side git is 2.15. The network topology is all on site - nothing ventures outside our firewall to the Big Bad Internet. It's fedora desktop to Centos server.

I'm at a loss. All straws gratefully grabbed.

[ttaylorr]

Hey @KC-SFC, thanks for opening this issue up, and sorry that you're having trouble!

Except that it randomly will throw a 401 status when asked for file. The git repo has about 100 binary files in the tree, and there's no telling when you'll get the 401. If you repeatedly pull, the client-side cache will eventually populate and the "error goes away" for the user.

This seems like a big clue to me. I'm wondering if there's either:

1. An issue with timing de-sync between the client and server (since lfs-test-server does not implement the new relative expires_in property), or...
2. An issue with a perculiar credential helper set-up on the client end not responding with correct or up-to-date credentials.

Do you have anything interesting going on that might pertain to number (2)?

[technoweenie]

Except that it randomly will throw a 401 status when asked for file. The git repo has about 100 binary files in the tree, and there's no telling when you'll get the 401. If you repeatedly pull, the client-side cache will eventually populate and the "error goes away" for the user.

What happens when you get a 401? Are you prompted to enter a user and pass? Does it go away for a little bit? If so, try upgrading to LFS v2.3.4. It caches those command prompts in memory for a single git lfs command, even if you have no credential helper configured.

Unfortunately, this is a test server that's only used as a reference implementation, and has never been run in a long term production scenario by any of the authors. I'd highly recommend upgrading to a self hosted Git installation with LFS support baked in. If not, maybe something like Artifactory would work better for you.

[KC-FLIR]

Hi Team - and I have to say a big THANK YOU for the hard work in the implementation of this test server code. I knew what I was doing when I used it - it does say in BIG letters "not for production". We don't use paid-for git hosting so yours was the only game in town for LFS, and it does work well; I hope you're proud of it; you should be. We've not lost a single file, we can get everything and it solves our what-do-you-do-with-binaries problem (I'm not keen on "use SVN" as the answer to that question)

The username an password for LFS is baked-in to URL in .lfsconfig i.e. 'https://someuser:somepassword@ourserver:9999", and AFAIK no credentials helper set up. We use ssh for our normal git clones, so don't need one to help authenticate for that; the joy of linux .ssh/config eh?

The 401 failure just results in our scripts failing on the clone. Run it again and the failing object comes down fine, with a 401 on another file in the updated list.

I've built the latest git for the clients and with sort of the latest LFS, so if that changes anything. The only place for a 401 is in the server.go code, L567 - just after the Header set which mentions the basic realm of git-lfs-server. Bearing in mind I've got no idea what this is doing (!) but is there an issue with us running 9 instances of git-lfs-server with the same realm name?

Or is it some problem with the meta store not returning the right data quickly enough?

[ttaylorr]

@KC-SFC Thanks, and you'll have to bear with me since it's been a little while since the last time I've looked through this code. The *MetaStore authenticates here:

https://github.com/git-lfs/lfs-test-server/blob/master/meta_store.go#L442

Which returns false if the password is the empty string,"". Can you double check that the user is sending the right password, and that is has length greater than zero?

[KC-FLIR]

That's a great question. The username and password length should be straight from the .lsconfig - which is committed into the repo. It should be the same on every call to the API.

I wonder if the IT networking team are messing with our traffic.

TY for the help. I have some digging to do.

[technoweenie]

If you want to confirm whether the Git LFS client is doing the right thing, run this:

$ LFS_DEBUG_HTTP=1 GIT_CURL_VERBOSE=1 git ...

This will add the full http request and responses to the command output. LFS_DEBUG_HTTP is a custom flag that enables LFS to dump the Authorization header too. If you're running a command like git pull, then you'll see Git's HTTP requests too. A command like git lfs pull should be enough to check just LFS.

[aggieNick02]

Stumbled here looking to solve https://github.com/git-lfs/git-lfs/issues/3416, which may be related. I assumed it was a git-lfs issue, but it may be an lfs-test-server issue.

We are also playing with using lfs-test-server in a production-ish environment (with daily backups to deal with failures). We use github.com, but it isn't a viable solution for our large files both because of the 2GB size limit and the massive time hit we would take uploading and downloading giant files over the internet. We don't want to do a self-hosted git, but maybe we have to consider it.