Known issues with git mirrors

[DrJosh9000]

This issue exists to billboard known problems with git mirrors across different agent versions.

Checkouts sometimes fail with 'bad object', 'did not send all necessary objects' errors

Issue: Checkouts sometimes fail because the mirror appears to contain the commit needed for a job, but the checkout phase fails with a 'bad object' or '{repo} did not send all necessary objects' error. As a recovery step the agent may try to fetch all refs, but still fails again. This could be due to corruption in the mirror repo, or due to objects removed normally from the mirror but still referenced by the checkout directory.

Affected versions: Unknown, possibly since v3.23.0.

Status: A workaround (#2286) is in v3.52.0.

Workarounds/debugging:

• Update to the latest agent release.
• Use the --clean-checkout flag or set the environment variable BUILDKITE_CLEAN_CHECKOUT=true.
• Delete the affected mirror directory, allowing the agent to make a fresh clone.
• Manually run git fsck --full in the checkout directory and manually remove any bad objects.
• In the mirror, manually run git fsck --full, remove bad objects, and then run git remote update to ensure the mirror is not corrupt.
• Disable git mirrors.

Mirrors of submodules not fetched

Issue: When the agent runs jobs that check out a repository that uses git submodules, the submodules should also be mirrored. The main repository mirror is updated (fetched) to ensure the commit or branch being built was present, but a bug meant that submodule mirrors was not fetched. As a result, submodule mirrors might become out -of-date, missing commits or objects needed by the main repo.

Affected versions: Between v3.44.0 and v3.49.0, inclusive.

Status: Fixed with #2203, which was released as part of v3.50.

Workarounds/debugging:

• Update to the latest agent release.
• If each host only runs one agent, include a manual git fetch origin for all submodule mirror directories in your pipeline.
• Use a custom checkout hook.
• Disable git mirrors.

Mirrors of submodules pull from wrong remote URL

Issue: When the agent runs jobs that check out a repository that uses git submodules, the submodules should also be mirrored. A bug caused the remote URL in the mirror directory for each submodule to be the wrong URL.

Affected versions: Between v3.44.0 and v3.48.0, inclusive.

Status: Fixed with #2144, but because the submodule mirror might not be re-fetched, it can still contain the wrong git history and objects. The missing fetch is fixed by #2203, which was released as part of v3.50.

Workarounds/debugging:

• Update to the latest agent release.
• If each host only runs one agent, update to at least agent v3.49 and include a manual git fetch origin for all submodule mirror directories in your pipeline.
• Use a custom checkout hook.
• Disable git mirrors.