atlantis
atlantis copied to clipboard
runatlantis
Atlantis fails to unlock if PR declined mid-plan
Atlantis appears to have an issue where if a PR is declined while a plan is in flight, the locks are not properly removed. Or perhaps they are being removed at the time of the declining, but as the plan is not aborted, new locks are perhaps being created. Either way I now have to go through the UI and turn off nearly 100 locks for a PR which is awful.
You might be able to reopen the PR and close it again to unlock (after plans) as a workaround..
when you close a PR in github the locks are removed. I use 0.15.0 and this works, what version @ryan-dyer-sp are you using?
Yes Atlantis will typically remove the locks when you close a PR. However, if atlantis is in the middle of performing a plan operation when the closing of the PR occurs, not all locks are cleaned up. We have seen this multiple times when someone submits a PR which result in 100s of project changes. They realize their mistake and close the PR while the plan is still in flight. And we're left with 100s of locks in the atlantis.db with no way to remove them. If #1393 were fixed, that would provide a workaround for many ways in which PRs can be closed but the locks still exist in the DB.
so your problem is only with atlantis locks but somehow you do not have issues with remote state locks in terraform? if I cancel an apply on terraform I end with the same problem.
No, my problem is not only with atlantis locks. The only problem I have I cant solve in less than a few seconds with some other workaround is with atlantis locks. The UI is fine if you only need to delete a lock or two. When you have a PR with over a hundred. Not so much. And when you have multiple PRs with dozens of locks and you only need to unlock those for a single PR. deleting the entire db, cleaning up the working dir, and restarting atlantis does not make for a decent workaround. Even though we've had to resort to it multiple times now.
and I'm guessing the unlock comment command does not work for you either?
I'm asking question to completely understand the issue and see how we could help you
This ticket was written before the unlock command existed. And the unlock command doesnt work once a PR is closed (#1393). As we use bitbucket, you cant reopen a PR once it is Declined in order to issue an unlock.
If #1393 were addressed, the issue with this ticket would be less of an issue as there would be a sufficient enough workaround that I wouldnt care about the various ways a PR can be closed and no way to unlock all of its projects other than through the UI.
@ryan-dyer-sp @jamengual I ran into this issue this past week. I can take a crack at it.
The issue is that the two "events" are unaware of each other. A command can come through either because of a GH command or PR closing. At that point in time it will execute and clean up any locks, however since a plan/apply is still running it will create more locks after thus the race condition.
There are two ways to solve this:
- Emit a signal to
project_command_pool_executorto exit early prior to cleaning up locks - Wait on firing the clean up locks event until the plan/apply finishes
@starkpister I'd imagine we'd be open to a PR for the unlock command to work on closed PRs. PRs welcome.
As great a solution as that might have been 6 months ago, we have since switched to a different method for deploying our terraform because of this bug.
Apologies. This project is maintained by open source developers and we welcome new changes to improve the project.
I think allowing the unlock command on closed PRs would be sufficient here.
Submitted a PR to allow commenting unlock on closed PRs https://github.com/runatlantis/atlantis/pull/2916 as suggested.
