Failure during plan with multiple workspaces with same name: deleting directory not empty

[andyshinn]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Plans using multiple projects with same same workspace name but different folder fail with:

deleting dir "/home/atlantis/.atlantis/repos/VHT/seabass/1904/govqa" before cloning: unlinkat /home/atlantis/.atlantis/repos/VHT/seabass/1904/govqa/.git/hooks: directory not empty

Reproduction Steps

Automatic plan via repository with two different project folders using same workspace name.

Logs

Logs

plan success. output available at: https://github.com/myorg/myapp/pull/1904
successfully ran ""/home/atlantis/.atlantis/bin/terraform1.1.4 plan -input=false -refresh -out \""/home/atlantis/.atlantis/repos/myorg/myapp/1904/prod/automation/terraform/jenkins/jenkins-prod-prod.tfplan\"""" in ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/prod/automation/terraform/jenkins""
plan success. output available at: https://github.com/myorg/myapp/pull/1904
successfully ran ""/home/atlantis/.atlantis/bin/terraform1.1.4 plan -input=false -refresh -out \""/home/atlantis/.atlantis/repos/myorg/myapp/1904/qa/automation/terraform/jenkins/jenkins-qa-qa.tfplan\"""" in ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/qa/automation/terraform/jenkins""
successfully ran ""/home/atlantis/.atlantis/bin/terraform1.1.4 workspace select prod"" in ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/prod/automation/terraform/jenkins""
successfully ran ""/home/atlantis/.atlantis/bin/terraform1.1.4 workspace show"" in ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/prod/automation/terraform/jenkins""
successfully ran ""/home/atlantis/.atlantis/bin/terraform1.1.4 init -input=false"" in ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/prod/automation/terraform/jenkins""
successfully ran ""/home/atlantis/.atlantis/bin/terraform1.1.4 workspace select qa"" in ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/qa/automation/terraform/jenkins""
successfully ran ""/home/atlantis/.atlantis/bin/terraform1.1.4 workspace show"" in ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/qa/automation/terraform/jenkins""
successfully ran ""/home/atlantis/.atlantis/bin/terraform1.1.4 init -input=false"" in ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/qa/automation/terraform/jenkins""
2022-05-24 20:40:19 UTC | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received
Error running plan operation: deleting dir ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/prod"" before cloning: unlinkat /home/atlantis/.atlantis/repos/myorg/myapp/1904/prod/.git/hooks: directory not empty
Error running plan operation: deleting dir ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/qa"" before cloning: unlinkat /home/atlantis/.atlantis/repos/myorg/myapp/1904/qa/.git/hooks: directory not empty
will re-clone repo, could not determine if was at correct commit: git rev-parse HEAD: exit status 128: fatal: not a git repository (or any parent up to mount point /home)Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
acquired lock with id ""myorg/myapp/automation/terraform/account/qa""
will re-clone repo, could not determine if was at correct commit: git rev-parse HEAD: exit status 128: fatal: not a git repository (or any parent up to mount point /home)Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
creating dir ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/qa""
acquired lock with id ""myorg/myapp/automation/terraform/jenkins/qa""
acquired lock with id ""myorg/myapp/automation/terraform/account/prod""
creating dir ""/home/atlantis/.atlantis/repos/myorg/myapp/1904/prod""
acquired lock with id ""myorg/myapp/automation/terraform/jenkins/prod""
Running plans in parallel

Environment details

If not already included, please provide the following:

• Atlantis version: v0.19.4-pre.20220513
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: No
• Atlantis flags:

Atlantis server-side config file:

gh-org: myorg
default-tf-version: v1.1.4
repos:
  - id: /.*/
    allowed_overrides:
      - apply_requirements
    apply_requirements:
      - approved
      - mergeable
      - undiverged
    workflow: default
metrics:
  statsd:
    host: "127.0.0.1"
    port: "8125"

Repo atlantis.yaml file:

version: 3
parallel_plan: true
parallel_apply: true
projects:
  - name: account-qa
    dir: automation/terraform/account
    workspace: qa
    autoplan:
      when_modified: ["*.tf", "../modules/**/*.tf", "modules/**/*.tf"]
      enabled: true
    apply_requirements: [mergeable, approved]
  - name: jenkins-qa
    dir: automation/terraform/jenkins
    workspace: qa
    autoplan:
      when_modified: ["*.tf", "../modules/**/*.tf", "modules/**/*.tf"]
      enabled: true
    apply_requirements: [mergeable, approved]

Additional Context

I thought it might be related to https://github.com/runatlantis/atlantis/pull/2180 but that PR appears to have been reverted in https://github.com/runatlantis/atlantis/pull/2253 so not sure.

[fblgit]

$workDir formulation... does not include $projectName. workDir is formulated before opening the config.yaml (where projectName is declared). Interim I would suggest to use different workspaces... even add some notes on the docs.

[snorlaX-sleeps]

@andyshinn with the current iteration of Atlantis you cannot have workspaces with the same name and do parallel plans / applies. The PR you linked (https://github.com/runatlantis/atlantis/pull/2180) fixes that and we are currently using that version of the build ourselves. There were apparently issues with the workspaces and custom pre-workflow-hooks, still unsure why it caused a revet