atlantis icon indicating copy to clipboard operation
atlantis copied to clipboard

Atlantis Apply is not accounting for missing plan files.

Open rawlbot opened this issue 3 years ago • 6 comments

Hi everyone. I'm seeing some strange behavior with Atlantis that I believe might be a bug. I'm running version 0.17.0.

We run Atlantis in containers and have no persistent storage available which means that if a PR sits open for too long there is a chance that someone has bounced the Atlantis pods and nuked any previously generated plan files. We are also using terragrunt-atlantis-config and pre workflow hooks which I believe is what exposes the bug (more on this near the bottom).

In the past (before we started using pre workflow hooks) missing plans due to pod bounces would manifest as such:

  • PR is opened and Atlantis auto plans and reports the following: This plan was saved to: /home/container/.atlantis/repos/IAC/repo-name/357/default/cloud/plan.tfplan

  • The PR sits for an hour or two while someone reviews and approves it. During this time the Atlantis pods are sometimes bounced and the plans previously created no longer exist.

  • The submitter, unaware of the pod bounce, would come back to the PR after getting approvals and run atlantis apply and then get this error: Apply Error stat /home/container/.atlantis/repos/IAC/repo-name/357: no such file or directory

Recently (after implementing pre workflow hooks and terragrunt-atlantis-config) Atlantis started being ok with missing plans and began reporting the following:

  • Ran Apply for 0 projects: Automatically merging because all plans have been successfully applied.

As you can imagine, my base branch is now out of sync with my state which requires additional PRs to bring them in sync.

My ability in go is elementary at best, but I spent some time digging into this and I believe the issue might be with how pending_plan_finder is implementing DefaultPendingPlanFinder. I suspect the bug is here is because it seems that the return value of plans can be empty, regardless if a plan previously existed for the PR.

This becomes problematic for us because I believe Atlantis is only checking for the existence of the working directory when trying to determine if it should proceed with applying and merging a previously planned PR which is a condition that will never be false now that pre workflow hooks run on every command and can generate this working directory ad hoc.

rawlbot avatar May 11 '21 01:05 rawlbot

We have a similar problem on a PR that has 0 planed changes. atlantis apply ends up with an error:

atlantis plan
Ran Plan for 0 projects:
atlantis apply
stat /home/atlantis/.atlantis/repos/my_org/my_repo/87: no such file or directory

Previously (0.16.1) it worked as expected and just reported:

Ran Apply for 0 projects:

That is pretty breaking change for us too...

Eddman avatar May 12 '21 14:05 Eddman

It seems that the issue we are running into is actually already reported in #1264. We are creating the conditions with terragrunt-atlantis-config by laying down the workspace dirs in the pre-workflow hooks but are ultimately dealing with missing locks since the plan files are gone.

rawlbot avatar May 19 '21 22:05 rawlbot

Any news here?

This is still a blocker for us. We are still at 0.16 due to this one.

Eddman avatar Jul 14 '21 12:07 Eddman

I don't understand why now in 0.17 no empty plan is generated as it was in 0.16...

Eddman avatar Jul 14 '21 12:07 Eddman

is this still an issue with v0.19.8?

jamengual avatar Aug 26 '22 02:08 jamengual

We are still seeing this with v0.19.8 if/when the locks are deleted from the underlying storage (i.e. no GH comments indicating the locks were deleted via the UI)

andrewjamesbrown avatar Sep 21 '22 15:09 andrewjamesbrown