Getting "fatal: loose object <id> is corrupt" errors during GitHub checkout

[EvHaus]

We're having some intermittent issues with our Buildkite GitHub integration. Some PR builds randomly fail with this:

Preparing working directory | 9s
-- | --
  | $ cd /var/buildkite/builds/bk-agent-agent-64b47594c8-pqd6l-1/myproject/myproject
  | # Host "github.com" already in list of known hosts at "/root/.ssh/known_hosts"
  | $ git remote set-url origin [email protected]:myproject/myproject.git
  | $ git clean -ffxdq
  | # Fetch and checkout pull request head from GitHub
  | $ git fetch -v --prune origin refs/pull/17681/head
  | error: object file .git/objects/7f/e9776a08bd8a1539f90d26e4f36dc1a1ede382 is empty
  | error: object file .git/objects/7f/e9776a08bd8a1539f90d26e4f36dc1a1ede382 is empty
  | fatal: loose object 7fe9776a08bd8a1539f90d26e4f36dc1a1ede382 (stored in .git/objects/7f/e9776a08bd8a1539f90d26e4f36dc1a1ede382) is corrupt
  | ⚠️ Warning: Checkout failed! exit status 128 (Attempt 1/3 Retrying in 2s)
  | # Host "github.com" already in list of known hosts at "/root/.ssh/known_hosts"
  | $ git remote set-url origin [email protected]:myproject/myproject.git
  | $ git clean -ffxdq
  | # Fetch and checkout pull request head from GitHub
  | $ git fetch -v --prune origin refs/pull/17681/head
  | error: object file .git/objects/7f/e9776a08bd8a1539f90d26e4f36dc1a1ede382 is empty
  | error: object file .git/objects/7f/e9776a08bd8a1539f90d26e4f36dc1a1ede382 is empty
  | fatal: loose object 7fe9776a08bd8a1539f90d26e4f36dc1a1ede382 (stored in .git/objects/7f/e9776a08bd8a1539f90d26e4f36dc1a1ede382) is corrupt
  | ⚠️ Warning: Checkout failed! exit status 128 (Attempt 2/3 Retrying in 2s)
  | # Host "github.com" already in list of known hosts at "/root/.ssh/known_hosts"
  | $ git remote set-url origin [email protected]:myproject/myproject.git
  | $ git clean -ffxdq
  | # Fetch and checkout pull request head from GitHub
  | $ git fetch -v --prune origin refs/pull/17681/head
  | error: object file .git/objects/7f/e9776a08bd8a1539f90d26e4f36dc1a1ede382 is empty
  | error: object file .git/objects/7f/e9776a08bd8a1539f90d26e4f36dc1a1ede382 is empty
  | fatal: loose object 7fe9776a08bd8a1539f90d26e4f36dc1a1ede382 (stored in .git/objects/7f/e9776a08bd8a1539f90d26e4f36dc1a1ede382) is corrupt
  | ⚠️ Warning: Checkout failed! exit status 128 (Attempt 3/3 Retrying in 2s)
  | 🚨 Error: exit status 128

This happens most commonly if you commit+push to a PR very quickly. The first commit Buildkite gets works, but the following ones fail with this error. However we've seen this happen ever on the first and only commit of a PR.

This failure happens in Buildkite's initial steps before it even gets to running any of our scripts or plugins, so we're not sure how to further debug it. We're on Buildkite Agent 3.22.0.

Could you help us understand what might be going wrong?

[jsoo1]

I have seen this, too. Is this due to restarts interrupting git operations?