akuity
git-push: concurrent promotions retry not working
Checklist
- [x] I've searched the issue queue to verify this is not a duplicate bug report.
- [x] I've included steps to reproduce the bug.
- [x] I've pasted the output of
kargo version. - [x] I've pasted logs, if applicable.
Description
It seems the built-in retry functionality is not working for git-push when two stages are trying to concurrently push to the same branch, but different file locations.
The docs state that the step should handle this scenario by doing a pull/rebase up to 50 times be default, but this does not seem to be happening.
No config options for the git-push step are being set other than:
- path
- targetBranch
Error:
step "task-1::do-push" met error threshold of 1: error running step "task-1::do-push": error pushing commits to remote: error pushing branch: error executing cmd [/usr/bin/git push origin HEAD:main]: To https://my-repo.git ! [remote rejected] HEAD -> main (incorrect old value provided) error: failed to push some refs to 'https://my-repo.git'
Screenshots
N/A
Steps to Reproduce
Using a remote GitLab project:
- Create two stages which run in parallel, writing to the same repo/branch, but to separate file locations.
- Trigger promotion for them both through a prior stage: "Promote to downstream"
- One of the stages successfully pushes, but the 2nd one fails with the above error.
Version
Kargo: v1.8.1
GitLab: 18.5
Logs
N/A
It's a long story, but anyone who's worked with Git via Go for long enough is familiar -- all our Git operations are actually implemented by exec'ing the git CLI. In the case of a push, there are a few different kinds of specific error text we look for in stderr as an indication that if we just pull/rebase again, the next attempt may be more successful. The message you're getting isn't one that's recognized, so you're getting an immediate failure instead of a retry.
[remote rejected] HEAD -> main (incorrect old value provided)
Probably we just need to update a regular expression that's used for identifying this scenario.
Before we do that... I believe the messages displayed in these cases originate server-side and are simply echoed back by the CLI. Would you be able to tell us what Git hosting provider you are using? It would help us to recreate this and verify the fix.
I've updated the issue with the GitLab version and title/steps to reflect that this is happening in GitLab.
Actually, @nitishfy if you are looking for something, this would be a good one. 🙏
This doesn't seem GitLab-specific, as it turns out.
I cannot reproduce this error, but my research suggests that this occurs not when one remote is simply behind another because of closely timed pushes, but rather when pushes that are truly concurrent are racing one another.
Copilot offered this scenario for illustrative purposes:
T1: Your client connects and asks "what refs do you have?" T2: Server responds: "main is at commit A" T3: Your client sends packfile with new commits T4: ⚡ ANOTHER PUSH completes, moving main to commit B ⚡ T5: Your client says "update main from A to C" T6: Server checks: "main is at B, not A - REJECTED"
We have similar issue with gitlab while multiple kargo deployments trying to push to same branch.
step "task-5::push" met error threshold of 1: error running step "task-5::push": error pushing commits to remote: error pushing branch: error executing cmd [/usr/bin/git push origin HEAD:dev]: remote: error: cannot lock ref 'refs/heads/dev': is at e15cfb28e46f3321221c7694973de90ffccb32b1 but expected a41aadc9aff7c8685e47de5f3b9dd0aa73fa6799 To https://xxxxxx.git ! [remote rejected] HEAD -> dev (failed to update ref) error: failed to push some refs to 'https://xxxxxx.git'
After some investigation it looks like it is related to regex as regex is expecting something like this:
! [remote rejected] HEAD -> experiment (cannot lock ref 'refs/heads/experiment'...)
but gitlab is returning:
! [remote rejected] HEAD -> dev (failed to update ref)
