kargo icon indicating copy to clipboard operation
kargo copied to clipboard
verified publisher icon akuity

git-clone step randomly fails with application-repository already exists

Open iplcmik opened this issue 2 months ago • 4 comments

Checklist

  • [x] I've searched the issue queue to verify this is not a duplicate bug report.
  • [x] I've included steps to reproduce the bug.
  • [x] I've pasted the output of kargo version.
  • [x] I've pasted logs, if applicable.

Description

Kargo promotion randomly fails with error below on git-clone step

step "step1" met error threshold of 1: error running step "step1": error adding work tree ./application-repository to repo https://code.rbi.tech/raiffeisen/mercury-certmanager-manifest.git: error adding working tree at "/tmp/promotion-c09adad7-a9e6-45de-a574-c3c5b9ef2ff6/application-repository": error executing cmd [/usr/bin/git worktree add /tmp/promotion-c09adad7-a9e6-45de-a574-c3c5b9ef2ff6/application-repository cdbee0f6bc1d937b09e51172fbc67104300a234d]: Preparing worktree (detached HEAD cdbee0f) fatal: '/tmp/promotion-c09adad7-a9e6-45de-a574-c3c5b9ef2ff6/application-repository already exists

Steps to Reproduce

Promote freight on stage . Stage first step configured:

    - as: step1
      config:
        checkout:
          - commit: ${{ commitFrom(vars.applicationRepoURL,
              warehouse(vars.warehouseName)).ID }}
            path: ./application-repository
        repoURL: ${{ vars.applicationRepoURL }}
      uses: git-clone

Repeat promotion few times. I observed tha it's reproduced when stage has 2 promotions, one created when the other is not finished yet

Version

v1.7.5

Logs

time="2025-10-24T09:59:17Z" level=debug msg="reconciling Promotion" namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4
time="2025-10-24T09:59:17Z" level=debug msg="Stage is not awaiting Promotion" freight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4 stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="syncing Promotions" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="syncing Freight" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="assessing health" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="Promotion is in progress: no health checks to perform" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="done reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="enqueued Promotion for reconciliation" namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4 stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="reconciling Promotion" namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4
time="2025-10-24T09:59:17Z" level=info msg="began promotion" freight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4 stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=error msg="Reconciler error" Stage="{mer22-dev-okd01 mercury-certmanager}" controller=stage controllerGroup=kargo.akuity.io controllerKind=Stage error="Stage health evaluated to Unknown" name=mer22-dev-okd01 namespace=mercury-certmanager reconcileID="\"e4a6ea4c-7f5b-4fb3-aa19-f935df83c2b4\""
time="2025-10-24T09:59:17Z" level=debug msg="reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="syncing Promotions" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="syncing Freight" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="assessing health" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="Promotion is in progress: no health checks to perform" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=debug msg="done reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:17Z" level=error msg="Reconciler error" Stage="{mer22-dev-okd01 mercury-certmanager}" controller=stage controllerGroup=kargo.akuity.io controllerKind=Stage error="Stage health evaluated to Unknown" name=mer22-dev-okd01 namespace=mercury-certmanager reconcileID="\"e2d6eacf-fc3b-4dc7-ad78-0e5bd8183e15\""
time="2025-10-24T09:59:19Z" level=debug msg="reconciling Promotion" namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath3873jdy5rsfzg9ntjy0.bd5cec4
time="2025-10-24T09:59:19Z" level=debug msg="Stage is not awaiting Promotion" freight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath3873jdy5rsfzg9ntjy0.bd5cec4 stage=mer22-dev-okd01
time="2025-10-24T09:59:19Z" level=debug msg="reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:19Z" level=debug msg="syncing Promotions" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:19Z" level=debug msg="syncing Freight" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:19Z" level=debug msg="assessing health" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:19Z" level=debug msg="Promotion is in progress: no health checks to perform" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:19Z" level=debug msg="done reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:19Z" level=error msg="Reconciler error" Stage="{mer22-dev-okd01 mercury-certmanager}" controller=stage controllerGroup=kargo.akuity.io controllerKind=Stage error="Stage health evaluated to Unknown" name=mer22-dev-okd01 namespace=mercury-certmanager reconcileID="\"9f314cb5-65b5-4898-9310-51cedac6990c\""
time="2025-10-24T09:59:21Z" level=debug msg="discovered commits" count=10 namespace=mercury-servicemesh warehouse=mer22-dev-okd01-wh
time="2025-10-24T09:59:21Z" level=debug msg="discovered commits" count=10 namespace=mercury-servicemesh repo="https://somegitrepo.com/mercury-manifest.git" warehouse=mer22-dev-okd01-wh
time="2025-10-24T09:59:21Z" level=debug msg="discovered latest artifacts" namespace=mercury-servicemesh warehouse=mer22-dev-okd01-wh
time="2025-10-24T09:59:21Z" level=debug msg="done reconciling Warehouse" namespace=mercury-servicemesh warehouse=mer22-dev-okd01-wh
time="2025-10-24T09:59:24Z" level=debug msg="reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:24Z" level=debug msg="syncing Promotions" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:24Z" level=debug msg="syncing Freight" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:24Z" level=debug msg="assessing health" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:24Z" level=debug msg="Promotion is in progress: no health checks to perform" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:24Z" level=debug msg="done reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:24Z" level=error msg="Reconciler error" Stage="{mer22-dev-okd01 mercury-certmanager}" controller=stage controllerGroup=kargo.akuity.io controllerKind=Stage error="Stage health evaluated to Unknown" name=mer22-dev-okd01 namespace=mercury-certmanager reconcileID="\"99ae1ba7-75a1-4c3e-a497-b11a4a27ed05\""
time="2025-10-24T09:59:27Z" level=debug msg="reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:27Z" level=debug msg="syncing Promotions" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:27Z" level=debug msg="syncing Freight" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:27Z" level=debug msg="assessing health" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:27Z" level=debug msg="Promotion is in progress: no health checks to perform" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:27Z" level=debug msg="done reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:27Z" level=error msg="Reconciler error" Stage="{mer22-dev-okd01 mercury-certmanager}" controller=stage controllerGroup=kargo.akuity.io controllerKind=Stage error="Stage health evaluated to Unknown" name=mer22-dev-okd01 namespace=mercury-certmanager reconcileID="\"1582151b-af46-4c3d-aab1-e03b7581996c\""
time="2025-10-24T09:59:28Z" level=debug msg="reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:28Z" level=debug msg="syncing Promotions" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:28Z" level=debug msg="syncing Freight" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:28Z" level=debug msg="assessing health" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:28Z" level=debug msg="Promotion is in progress: no health checks to perform" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:28Z" level=debug msg="done reconciling Stage" controlFlow=false namespace=mercury-certmanager stage=mer22-dev-okd01
time="2025-10-24T09:59:28Z" level=error msg="Reconciler error" Stage="{mer22-dev-okd01 mercury-certmanager}" controller=stage controllerGroup=kargo.akuity.io controllerKind=Stage error="Stage health evaluated to Unknown" name=mer22-dev-okd01 namespace=mercury-certmanager reconcileID="\"69dfbedf-7220-4160-a8a2-3ff87cd864d7\""
time="2025-10-24T09:59:35Z" level=debug msg="reconciling Warehouse" namespace=mercury-crossplane-lib warehouse=mer22-dev-okd01-aws-sns-wh
time="2025-10-24T09:59:35Z" level=debug msg="obtained credentials for git repo" namespace=mercury-crossplane-lib repo="https://somegitrepo.com/mercury-crossplane-lib.git" warehouse=mer22-dev-okd01-aws-sns-wh
time="2025-10-24T09:59:35Z" level=debug msg="cloning repository" namespace=mercury-crossplane-lib warehouse=mer22-dev-okd01-aws-sns-wh
time="2025-10-24T09:59:35Z" level=debug msg="discovered commits" count=10 namespace=mercury-crossplane-lib warehouse=mer22-dev-okd01-aws-sns-wh
time="2025-10-24T09:59:35Z" level=debug msg="discovered commits" count=10 namespace=mercury-crossplane-lib repo="https://somegitrepo.com/mercury-crossplane-lib.git" warehouse=mer22-dev-okd01-aws-sns-wh
time="2025-10-24T09:59:35Z" level=debug msg="obtained credentials for git repo" namespace=mercury-crossplane-lib repo="https://somegitrepo.com/mercury-manifest.git" warehouse=mer22-dev-okd01-aws-sns-wh
time="2025-10-24T09:59:35Z" level=debug msg="cloning repository" namespace=mercury-crossplane-lib warehouse=mer22-dev-okd01-aws-sns-wh
time="2025-10-24T09:59:35Z" level=debug msg="executing argocd-update promotion step" freight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4 stage=mer22-dev-okd01
time="2025-10-24T09:59:35Z" level=debug msg="patched Argo CD Application" app=mercury-certmanager freight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4 stage=mer22-dev-okd01
time="2025-10-24T09:59:35Z" level=debug msg="done executing argocd-update promotion step" freight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4 stage=mer22-dev-okd01
time="2025-10-24T09:59:35Z" level=debug msg=promotion freight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea namespace=mercury-certmanager phase="\"Running\"" promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4 stage=mer22-dev-okd01 targetFreight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea
time="2025-10-24T09:59:35Z" level=debug msg="reconciling Promotion" namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4
time="2025-10-24T09:59:35Z" level=debug msg="continuing Promotion" freight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4 stage=mer22-dev-okd01
time="2025-10-24T09:59:36Z" level=error msg="error executing Promotion" error="step \"step1\" met error threshold of 1: error running step \"step1\": error adding work tree ./application-repository to repo https://somegitrepo.com/mercury-certmanager-manifest.git: error adding working tree at \"/tmp/promotion-c09adad7-a9e6-45de-a574-c3c5b9ef2ff6/application-repository\": error executing cmd [/usr/bin/git worktree add /tmp/promotion-c09adad7-a9e6-45de-a574-c3c5b9ef2ff6/application-repository cdbee0f6bc1d937b09e51172fbc67104300a234d]: Preparing worktree (detached HEAD cdbee0f)\nfatal: '/tmp/promotion-c09adad7-a9e6-45de-a574-c3c5b9ef2ff6/application-repository' already exists\n" freight=bd5cec4d13eec0eec7ee0374d88f8cb6c97ce8ea namespace=mercury-certmanager promotion=mer22-dev-okd01.01k8ath227kpjaz162hp03tz5g.bd5cec4 stage=mer22-dev-okd01

iplcmik avatar Oct 24 '25 07:10 iplcmik

I'm hitting the same issue with this config to create the branch

    - as: clone
      config:
        checkout:
          - branch: main
            create: true
            path: ./out
        repoURL: ${{ vars.gitRepoManifests }}
      continueOnError: false
      if: ""
      uses: git-clone

kirrmann avatar Oct 28 '25 09:10 kirrmann

This has been reported many times before. If you search, I'm sure you'll find many issues related to it. The git-clone step itself is not the culprit, rather this is a symptom of the entire promotion restarting from step 0 for some reason we've been struggling to identify. Posting the full YAML of your promotion, both spec and status, with necessary redactions might add valuable data to this ongoing bug hunt.

krancour avatar Oct 28 '25 19:10 krancour

I am pinning this issue for visibility in the hopes that:

  1. We'll get fewer duplicate reports of this.

  2. People will more quickly come to understand what the actual problem is. (For this reason, I'm leaving the title unchanged so that people will find it more easily.)

  3. We might finally get details from the community that help us nail down the root cause of Promotions mysteriously restarting without an obvious explanation (like the controller having restarted).

krancour avatar Oct 30 '25 18:10 krancour

Thanks @krancour . Do you have any recommendations for debug messages etc to change to try to get an idea of where it's panicking? Happy to add more logging with a build in prod but a few pointers would be helpful.

shamsalmon avatar Nov 03 '25 10:11 shamsalmon

I experienced the same issue. In my case, the root cause was that kargo-controller crashed during promotion with exit code 137 (OOMKilled). I increased the memory limit for Kargo, which resolved the issue.

P4sca1 avatar Dec 18 '25 09:12 P4sca1