Support git LFS

[oschwald]

What would you like Renovate to be able to do?

Use renovate with files stored in git lfs.

We use Yarn's offline-mirror feature to vendor the packages. The vendored packages are stored in Git using LFS. When Renovate tries to push the changes, it fails with the message below.

Relevant debug logs

Slightly sanitized error message.

DEBUG: 35 file(s) to commit (repository=group/repo, branch=renovate/js)
DEBUG: Committing files to branch renovate/js (repository=group/repo, branch=renovate/js)
DEBUG: Error commiting files (repository=group/repo, branch=renovate/js)
       "err": {
         "task": {
           "commands": [
             "push",
             "origin",
             "renovate/js:renovate/js",
             "--force",
             "-u",
             "--no-verify",
             "--verbose",
             "--porcelain"
           ],
           "concatStdErr": true,
           "format": "utf-8"
         },
         "message": "Pushing to https://github.company.com/group/repo.git\nPOST git-receive-pack (12865 bytes)\nremote: error: GH008: Your push referenced at least 24 unknown Git LFS objects:        \nremote:     fa64832d250b8a4beea35965a3bb9ab26bddfeb73dbfe836ed61e70e3974fe18        \nremote:     efc11cec699211605376abf2a7aa47b7c7e1854d4428bcef9e67b258918d64d5        \nremote:     8e92658ac0ed2e085bbf11f21bb455d6a9eeae173de21d376e9bbd7105378f1e        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.company.com/group/repo.git'\n", "stack": "Error: Pushing to https://github.company.com/group/repo.git\nPOST git-receive-pack (12865 bytes)\nremote: error: GH008: Your push referenced at least 24 unknown Git LFS objects:        \nremote:     fa64832d250b8a4beea35965a3bb9ab26bddfeb73dbfe836ed61e70e3974fe18        \nremote:     efc11cec699211605376abf2a7aa47b7c7e1854d4428bcef9e67b258918d64d5        \nremote:     8e92658ac0ed2e085bbf11f21bb455d6a9eeae173de21d376e9bbd7105378f1e        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.company.com/group/repo.git'\n\n    at GitExecutorChain.onFatalException (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:60:87)\n    at GitExecutorChain.<anonymous> (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:51:28)\n    at Generator.throw (<anonymous>)\n    at rejected (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at process._tickCallback (internal/process/next_tick.js:68:7)" }
DEBUG: Passing repository-changed error up (repository=group/repo, branch=renovate/js)
 INFO: Repository has changed during renovation - aborting (repository=group/repo)

[viceice]

For this we would need to make the renovate git layer git-lfs aware.

[viceice]

@oschwald Can you prepare a minimal public reproduction repo? So we have something to test against it.

[oschwald]

Thanks. I made an example repo at https://github.com/oschwald/renovate-lfs-issue.

With this repo, the exact failure is:

DEBUG: Error commiting files (repository=oschwald/renovate-lfs-issue, branch=renovate/js)
       "err": {
         "task": {
           "commands": [
             "push",
             "origin",
             "renovate/js:renovate/js",
             "--force",
             "-u",
             "--no-verify",
             "--verbose",
             "--porcelain"
           ],
           "concatStdErr": true,
           "format": "utf-8"
         },
         "message": "Pushing to https://github.com/oschwald/renovate-lfs-issue.git\nPOST git-receive-pack (4516 bytes)\nremote: error: GH008: Your push referenced at least 11 unknown Git LFS objects:        \nremote:     26acb6450ef03e44b1e1df2457588c0177013c9a6ded0f8e058c5d034a5391e2        \nremote:     47ab5c4571504bdee569f03e3423af5b51aa17d6a94866ddcae353ed2d9033eb        \nremote:     c609324ab889515f2f7354ddcc319b6080c9b76f2ac1441c03da031c85458696        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.com/oschwald/renovate-lfs-issue.git'\n",
         "stack": "Error: Pushing to https://github.com/oschwald/renovate-lfs-issue.git\nPOST git-receive-pack (4516 bytes)\nremote: error: GH008: Your push referenced at least 11 unknown Git LFS objects:        \nremote:     26acb6450ef03e44b1e1df2457588c0177013c9a6ded0f8e058c5d034a5391e2        \nremote:     47ab5c4571504bdee569f03e3423af5b51aa17d6a94866ddcae353ed2d9033eb        \nremote:     c609324ab889515f2f7354ddcc319b6080c9b76f2ac1441c03da031c85458696        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.com/oschwald/renovate-lfs-issue.git'\n\n    at GitExecutorChain.onFatalException (/home/greg/MaxMind/renovate-test/node_modules/simple-git/src/lib/runners/git-executor-chain.js:60:87)\n    at GitExecutorChain.<anonymous> (/home/greg/MaxMind/renovate-test/node_modules/simple-git/src/lib/runners/git-executor-chain.js:51:28)\n    at Generator.throw (<anonymous>)\n    at rejected (/home/greg/MaxMind/renovate-test/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"
       }

[github-actions[bot]]

Hi there,

The Renovate team needs your help! Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on minimal reproductions to understand what is needed.

We may close the issue if you have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[viceice]

I think this needs a new repo config options to configure files (minimatch / wildcard) to checkout with git lfs. By default git lfs should be disabled.

[github-actions[bot]]

Thank you for providing a reproduction! :tada: :rocket:

The Renovate team will take a look at the reproduction repository. Once we confirm the provided repository reproduces the problem, the label will be changed to reproduction:confirmed.

[sgtsquiggs]

git lfs uses a pre-push git hook, which is disabled by renovate using --no-verify #6521

[amacneil]

Hi friends,

What would it take to get git lfs pull support over the finish line? Can we contribute anything to help?

This is the one feature preventing us from using renovate, and like many others we are on yarn 2 with the yarn binary stored in lfs.

Unfortunately, dependabot doesn’t support yarn 2, and renovate doesn’t support git lfs, so we’re in a bit of a rock and hard place.

[jtbandes]

Hi @rarkins – any chance you can comment on the above? 💕

[rarkins]

@fgblomqvist @viceice could either of you summarize the current status?

[viceice]

We still need to define if it will be always enabled? That way i assume it will always automatically checkout all lfs files. So we don't like to have that.

I think we need to add lfs install to buildpack without full enable it. So install binaries only.

Then we need a configurable list of lfs files, which renovate should checkout from / commit to branch.

[amacneil]

Does anyone need this feature for anything other than yarn 2?

I think it would be sufficient for 99% of users to checkout a default list of files, e.g. git lfs pull --include .yarn yarn.lock **/package-lock.json.

If anyone arrives after that with a request for different files, then it could be made configurable (or just add to the lfs checkout list). I haven't heard any use case for making it configurable yet, but many people (like me) are stuck because of yarn 2.

[viceice]

Instead of fully disabling git lfs by default i like to now some common lfs.fetchexclude and /or lfs.fetchinclude settings for renovate to make this work for most users.

I plan to make those settings user configurable.

[amacneil]

Sounds great - looking forward to it!

[viceice]

So renovate defaults would be:

{
  allowGitLfs: true,
  gitLfsInclude: '.yarn,yarn.lock,**/package-lock.json',
  gitLfsExclude: undefined,
}

allowGitLfs: false would override includes to undefiend and excludes to *.

Any other files to consider by default?

[amacneil]

I think those will cover our use case.

Maybe call the flag enableGitLfs: true or even just gitLfs: true ?

[rarkins]

I'm all for letting self-hosted users get their hands on this first, and we work out how to handle it in the "untrusted" app environment later.

[amacneil]

Is it hard to get working in the shared environment? This doesn't seem like it should be any more or less safe than a regular git checkout.

[viceice]

@amacneil The hosted app has disabled git hooks for security concerns, so yes, it's not as easy to enable lfs there.

[defunctzombie]

Is there still a plan to enable git lfs support for the hosted app? Or is the recommendation to self-host?

[viceice]

Yes, but still needs some time for all preparations.

[gjasny]

FYI: buildpack git-lfs install is broken: https://github.com/containerbase/buildpack/pull/238

[steschi]

What's the current status? What is blocking this? I think yarn-v3 and zero-installs with git-lfs are getting more common but without proper lfs support (at least opt-in or only on self-hosted) these repositories can not take advantage of renovate at all.

[simon-liebehenschel]

It took some working hours for me to setup Renovate with git-lfs support (and a custom Python version), so I would like to share a working configuration with other people.

1. Build a custom Docker image.

   FROM renovate/renovate:34.24
   SHELL ["/bin/bash", "-o", "pipefail", "-c"]
   RUN install-tool python 3.10.8  \
       && install-tool poetry 1.2.2
   USER root
   RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash  \
       && apt-get install -y --no-install-recommends git-lfs=3.2.0 \
       && apt-get -y autoremove && apt-get clean && rm -rf /var/lib/apt/lists/*
   USER 1000
   

   Note that I install a Python version that I use. If you are fine to use the latest Python (Python 3.11), then change this Dockerfile. Calling USER root is required before calling apt-get or the command will fail due to the lack of permissions. If you do not need custom Python, etc. then just install git-lfs:

   FROM renovate/renovate:34.24
   SHELL ["/bin/bash", "-o", "pipefail", "-c"]
   USER root
   RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash  \
       && apt-get install -y --no-install-recommends git-lfs=3.2.0 \
       && apt-get -y autoremove && apt-get clean && rm -rf /var/lib/apt/lists/*
   USER 1000
   

2. Set required settings to support git-lfs.

   # This is mandatory for git-lfs or it will fail with the `GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all"`
   RENOVATE_GIT_NO_VERIFY: "[]"
   # Perhaps `RENOVATE_BINARY_SOURCE` is redundant if you already installed all required packages in your custom Docker image.
   RENOVATE_BINARY_SOURCE: "install"
   

   Other configuration options are omitted for the sake of brevity.

This Docker image and config were tested at GitLab SaaS.

[jtbandes]

FYI, for those using yarn, dependabot now supports yarn berry and git lfs (https://github.com/dependabot/dependabot-core/pull/5833).

[rarkins]

@AIGeneratedUsername thanks for posting your solution. I have some questions on some of the things you added:

RUN install-tool python 3.10.8  \
    && install-tool poetry 1.2.2

I'm wondering why these are necessary. Shouldn't both of these be dynamically installed due to binarySource=install at runtime?

RENOVATE_BINARY_SOURCE: "install"

First of all, I'm not sure if this is mandatory for Git LFS in any way. But also, it is now the default behavior for renovate/renovate non-slim images anyway.

The main one of interest to me is:

RENOVATE_GIT_NO_VERIFY: "[]"

Here is the option for this:

  {
    name: 'gitNoVerify',
    description:
      'Which Git commands will be run with the `--no-verify` option.',
    type: 'array',
    subType: 'string',
    allowString: true,
    allowedValues: ['commit', 'push'],
    default: ['commit', 'push'],
    stage: 'global',
    globalOnly: true,
  },

So by setting this you're making sure Renovate runs e.g. git push --no-verify. Is this may be a workaround and not yet a solution?

[viceice]

disable git no-verify is required because LFS is working via git hooks

[rarkins]

Would we have any way in future to detect LFS is in use and remove --no-verify automatically?

[viceice]

not easily, we also set it to suppress arbitrary git hooks from repos for security reasons.

[mat813]

I started using SheetJS in a projet, and they recommend vendoring the package file because the npm registry has an outdated one, as it is a bit large, I used git lfs to store it, and now renovate cannot run yarn install, it errors out with:

error Extracting tar content of undefined failed, the file appears to be corrupt: "Unexpected end of data"

Also, we're considering moving from yarn classic to yarn v3 using zero installs which kinda requires git-lfs if you don't want your repository size to explode. But we are blocked by renovate not supporting lfs.