Git mirrors appears to break git commands

[grrowl]

For the following pipeline.yml:

  - name: ': sadpanda:'
    command: git diff --quiet
    plugins:
      docker-compose#v3.0.3:
        run: tests
        config: .buildkite/docker-compose.yml
        volumes:
          - ${BUILDKITE_AGENT_BINARY_PATH:-/usr/bin/buildkite-agent}:/usr/bin/buildkite-agent

I get this output and error in buildkite:

$ docker-compose -f .buildkite/docker-compose.yml -p buildkite823f48f5b44c4a699c0c26227b09fb01 -f docker-compose.buildkite-7387-override.yml run --name buildkite823f48f5b44c4a699c0c26227b09fb01_tests_build_7387 -v /usr/bin/buildkite-agent:/usr/bin/buildkite-agent --rm tests /bin/sh -e -c 'git diff --quiet'
--
  | error: object directory /var/lib/buildkite-agent/git-mirrors/git-github-com-covidence-app-git/objects does not exist; check .git/objects/info/alternates.
  | fatal: unable to read 2d4d6566de832488614e81e72b3f9ff40af7e5be

Adding a volume mount to /var/lib/buildkite-agent to pipeline.yml fixes it:

  - name: ':parrot:'
    command: git diff --quiet
    plugins:
      docker-compose#v3.0.3:
        run: tests
        config: .buildkite/docker-compose.yml
        volumes:
          - ${BUILDKITE_AGENT_BINARY_PATH:-/usr/bin/buildkite-agent}:/usr/bin/buildkite-agent
          - /var/lib/buildkite-agent:/var/lib/buildkite-agent

$ docker-compose -f .buildkite/docker-compose.yml -p buildkite3981094a0b0547b9bc7fd436813e76c7 -f docker-compose.buildkite-7388-override.yml run --name buildkite3981094a0b0547b9bc7fd436813e76c7_tests_build_7388 -v /usr/bin/buildkite-agent:/usr/bin/buildkite-agent -v /var/lib/buildkite-agent:/var/lib/buildkite-agent --rm tests /bin/sh -e -c 'git diff --quiet'
--

[lox]

That will be because the mirrors dir isn't being mounted through (as you discovered).

This is probably an agent bug vs a docker-compose one, but I'm sorry it bit you!

The answer is to add the --dissociate option to git-clone-flags I believe. We'll make this change upstream in a future agent version.

[evandam]

Hey @lox, sorry to raise this from the grave but I'm looking for a cleaner solution to this. Previously I've mounted the mirrors dirs left and right across pipelines, but it would be great to avoid the need to, especially since most folks writing pipelines don't know all the ins and outs of Buildkite's configs.

Let me know if there's a better place to post this, but here's what I have so far.

My best guess is that the git-clone-mirror-flags applies to when Buildkite creates the mirror, not necessarily when it clones from it? In that case I suppose --dissociate should actually be set on git-clone-flags instead?

BUILDKITE_GIT_CLONE_MIRROR_FLAGS=-v --mirror --dissociate

pipeline.yml:

steps:
  - plugins:
      - docker#v3.5.0:
          image: alpine/git
          entrypoint: sh
          command: ["-c", "echo foo >> README.md && git diff origin/master"]

Preparing working directory

~~~ Preparing working directory
[90m# Creating "/var/lib/buildkite-agent/builds/buildkite-i-00257661cd4d89937-shared-usw2-ci1-9/wonolo/ops-debug"[0m
[90m$[0m cd /var/lib/buildkite-agent/builds/buildkite-i-00257661cd4d89937-shared-usw2-ci1-9/wonolo/ops-debug
[90m# Host "github.com" already in list of known hosts at "/var/lib/buildkite-agent/.ssh/known_hosts"[0m
[90m# Using git-mirrors experiment 🧪[0m
[90m$[0m cd /var/lib/buildkite-agent/buildkite-git-mirrors
[90m# Cloning a mirror of the repository to "/var/lib/buildkite-agent/buildkite-git-mirrors/git-github-com-Wonolo-ops-debug-git"[0m
[90m$[0m git clone -v --mirror --dissociate -- [email protected]:Wonolo/ops-debug.git /var/lib/buildkite-agent/buildkite-git-mirrors/git-github-com-Wonolo-ops-debug-git
Cloning into bare repository '/var/lib/buildkite-agent/buildkite-git-mirrors/git-github-com-Wonolo-ops-debug-git'...

remote: Enumerating objects: 65, done.[K

remote: Counting objects:   1% (1/65)[K
...
remote: Counting objects: 100% (65/65), done.[K

remote: Compressing objects:   2% (1/41)[K
...
remote: Compressing objects: 100% (41/41), done.[K

Receiving objects:   0% (1/1572)   
... 
Receiving objects:  99% (1557/1572)   
remote: Total 1572 (delta 37), reused 48 (delta 24), pack-reused 1507[K

Receiving objects: 100% (1572/1572)   
Receiving objects: 100% (1572/1572), 296.92 KiB | 817.00 KiB/s, done.

Resolving deltas:   0% (0/809)   
...
Resolving deltas: 100% (809/809), done.

[90m$[0m cd /var/lib/buildkite-agent/builds/buildkite-i-00257661cd4d89937-shared-usw2-ci1-9/wonolo/ops-debug
[90m$[0m git clone -v --reference /var/lib/buildkite-agent/buildkite-git-mirrors/git-github-com-Wonolo-ops-debug-git -- [email protected]:Wonolo/ops-debug.git .
Cloning into '.'...

[90m$[0m git clean -fxdq
[90m# Fetch and checkout remote branch HEAD commit[0m
[90m$[0m git fetch -v --prune origin master
From github.com:Wonolo/ops-debug

 * branch            master     -> FETCH_HEAD

 = [up to date]      master     -> origin/master

[90m$[0m git checkout -f FETCH_HEAD
Note: checking out 'FETCH_HEAD'.



You are in 'detached HEAD' state. You can look around, make experimental

changes and commit them, and you can discard any commits you make in this

state without impacting any branches by performing another checkout.



If you want to create a new branch to retain commits you create, you may

do so (now or later) by using -b with the checkout command again. Example:



  git checkout -b <new-branch-name>



HEAD is now at 4176a1d bump version 1.0.4

[90m# Cleaning again to catch any post-checkout changes[0m
[90m$[0m git clean -fxdq
[90m# Using resolve-commit-after-checkout experiment 🧪[0m
[90m# Updating BUILDKITE_COMMIT from "HEAD" to "4176a1dc626d18f70d6e163f56db289cb38c6406"[0m
[90m# Checking to see if Git data needs to be sent to Buildkite[0m
[90m$[0m buildkite-agent meta-data exists buildkite:git:commit
[90m# Sending Git commit information back to Buildkite[0m
[90m$[0m buildkite-agent meta-data set buildkite:git:commit "commit 4176a1dc626d18f70d6e163f56db289cb38c6406\nAuthor:     Evan Van Dam <[email protected]>\nAuthorDate: Sat Jul 25..."

Running command in alpine/git

[90m$[0m docker run -it --rm --init --volume /var/lib/buildkite-agent/builds/buildkite-i-00257661cd4d89937-shared-usw2-ci1-9/wonolo/ops-debug:/workdir --workdir /workdir --env BUILDKITE_JOB_ID --env BUILDKITE_BUILD_ID --env BUILDKITE_AGENT_ACCESS_TOKEN --volume /usr/bin/buildkite-agent:/usr/bin/buildkite-agent --entrypoint sh --label com.buildkite.job-id=40cbe93c-871d-48b6-bec4-8def1fe8cbdc alpine/git -c echo\ foo\ \>\>\ README.md\ \&\&\ git\ diff\ origin/master 

Unable to find image 'alpine/git:latest' locally

latest: Pulling from alpine/git

8c410e94e9f3: Pull complete 
[1BDigest: sha256:076ac2e35b474c2960785895cc4375303ab8eca37c139fb66977400598ab4989

Status: Downloaded newer image for alpine/git:latest

error: object directory /var/lib/buildkite-agent/buildkite-git-mirrors/git-github-com-Wonolo-rds-snapshotter-git/objects does not exist; check .git/objects/info/alternates

fatal: bad object origin/master

[31m🚨 Error: The command exited with status 128[0m

[ryansdwilson]

Hi, Has anyone figured out how to deal with this? we're trying to do a git diff in our pipeline run to determine changes vs master but the git mirrors breaks the ability to run the git command. I've tried adding the command below to our agent bootstrap script but still having trouble.

export BUILDKITE_GIT_CLONE_MIRROR_FLAGS="-v --mirror --dissociate"

[toote]

After reading and re-reading the description of this issue, the logs and related links, my understanding indicates that this was an issue with the agent that can be solved, as @lox mentioned by adding -dissociate to the agent's clone flags.

With that said, I don't think that there is a bug or something to do in this plugin though, specially 3+ years later.