docker-agent
docker-agent copied to clipboard
jenkinsci
Don't rely on linux distribution's package for `git-lfs`
Related:
* [New release to remediate CVEs for compliance git-lfs/git-lfs#6051](https://github.com/git-lfs/git-lfs/issues/6051)Note: this PR is only for Windows. Currently we rely on each Linux distribution's package for git-lfs for Linux images
Originally posted by @dduportal in https://github.com/jenkinsci/docker-agent/issues/1007#issuecomment-3012298815
Just tested building a debian image, it includes git-lfs 3.3.0-1+deb12u1, lagging behind official releases. (3.7.0 currently)
WDYT about having a distinct installation for ~~git and~~ git-lfs to ensure which versions of them are included in Linux images instead of relying on distribution packaging (and from which we don't know in advance which versions they're including)?
For git-lfs it is easy to contribute: we already have a specific installation on jenkinsci/docker using the tar gz distribution: https://github.com/jenkinsci/docker/blob/ba178d816f63543645d27f90638c5d68dc6037c0/debian/bookworm/hotspot/Dockerfile#L57-L65
It's "easy" because it works out of the box as a "statically" compiled binary (in theory: I haven't checked the dynamic library links neither the compilation flags).
For git it is not that simple: usually Git has dependencies with some distribution library to support cryptography, secrets managements and many other things.
There are (many good) reasons why distributions keep maintaining their own package of tools and are providing backports (for security and some feature). We might not want to create "frankeinstein" images with only a partial subset of features to provide for end users.
I knew there was a reason for git 😅
Editing my issue to only include git-lfs.
My experience matches with @dduportal .
The git-lfs project knows that their binary will be installed in many locations with many different git versions and they have successfully kept it portable to multiple git versions, multiple operating systems, and multiple operating system versions.
The git project has a different expectation that is better suited to operating systems configuring, building, and packaging git for that specific operating system.
Thanks for focusing this issue on git-lfs. I think that is the right choice.
PR finally ready for review: https://github.com/jenkinsci/docker-agent/pull/1009
Got a proper git-lfs test prepared, which list current git-lfs tracked files of a GitHub repository instead of just relying on git lfs version which might not be enough to ensure compatibility of a git-lfs release with the distributions and architectures we're delivering.
See this PR in draft:
- https://github.com/jenkinsci/docker-ssh-agent/pull/557
It does so by cloning https://github.com/lemeurherve/git-lfs-testing-repo/ then git lfs ls-files its "large" git-lfs tracked file (of 17 bytes).
This repository is from my own GitHub (free) account to avoid any potential billing on @jenkinsci or @jenkins-infra. The only downside when reaching the free git-lfs quota of 10Go: not be able to push new files back up.
A new GitHub organisation dedicated to hosting this repository can be setup if prefered over using mine.
WDYT @jenkinsci/team-docker-packaging?
Refs:
- https://docs.github.com/en/billing/concepts/product-billing/git-lfs#free-use-of-git-lfs
- https://docs.github.com/en/billing/concepts/product-billing/git-lfs#using-more-than-your-included-quota
- https://matrix.to/#/!CXfiHlEKGLXlkMPLEN:gitter.im/$w3lulJOkvfoFOOSXT6MboKHgb2wvjhUW4Mz4ZDcgK40?via=gitter.im&via=matrix.org&via=nani.wtf (thread on #jenkinsci/docker)
FWIW, I switched the test in https://github.com/jenkinsci/docker-ssh-agent/pull/557 to use git-lfs env (and not git-lfs version which is just outputing a value) to not rely on an external repository: https://github.com/jenkinsci/docker-ssh-agent/pull/557/commits/c278a1157107a6ec34bbd0295d3a50496ef5a9ee
As an alternative to a repository in an external GitHub organisation, I also thought about putting this testing repo in @jenkinsci and archive it to avoid abuse.
We could also try to get a git-lfs server (https://github.com/git-lfs/git-lfs/wiki/Implementations) running in tests, but then it's more work, and git-lfs env might be enough.
FTR, here is the output of git-lfs env from an agent image, checking various elements:
$ docker run -it --entrypoint bash jenkins/agent
jenkins@08ac0b7ad309:~$ git lfs env
git-lfs/3.7.0 (GitHub; linux arm64; go 1.24.4; git 92dddf56)
git version 2.47.3
LocalWorkingDir=
LocalGitDir=
LocalGitStorageDir=
LocalMediaDir=lfs/objects
LocalReferenceDirs=
TempDir=lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneVerifyUnreachableAlways=false
PruneRemoteName=origin
LfsStorageDir=lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/usr/lib/git-core
git config filter.lfs.process = ""
git config filter.lfs.smudge = ""
git config filter.lfs.clean = ""
FTR, here is the output of
git-lfs envfrom an agent image, checking various elements:$ docker run -it --entrypoint bash jenkins/agent jenkins@08ac0b7ad309:~$ git lfs env git-lfs/3.7.0 (GitHub; linux arm64; go 1.24.4; git 92dddf56) git version 2.47.3 LocalWorkingDir= LocalGitDir= LocalGitStorageDir= LocalMediaDir=lfs/objects LocalReferenceDirs= TempDir=lfs/tmp ConcurrentTransfers=8 TusTransfers=false BasicTransfersOnly=false SkipDownloadErrors=false FetchRecentAlways=false FetchRecentRefsDays=7 FetchRecentCommitsDays=0 FetchRecentRefsIncludeRemotes=true PruneOffsetDays=3 PruneVerifyRemoteAlways=false PruneVerifyUnreachableAlways=false PruneRemoteName=origin LfsStorageDir=lfs AccessDownload=none AccessUpload=none DownloadTransfers=basic,lfs-standalone-file,ssh UploadTransfers=basic,lfs-standalone-file,ssh GIT_EXEC_PATH=/usr/lib/git-core git config filter.lfs.process = "" git config filter.lfs.smudge = "" git config filter.lfs.clean = ""
That looks a good enough test for the "linux installation method change + 3.7.1 bump" 👏