Run plan/apply for multiple projects in parallel

[lkysow]

Via @psalaberria002, would like Atlantis to run plan and apply in parallel when running a command that needs to run on multiple projects. This would make things much faster.

[jolexa]

We have some repos that works on >10 AWS accounts (each account is a project in atlantis.yaml). When making a change to all the accounts, Atlantis is painfully show while it sequentially works on every account. We expect this usage to increase, by the way.

[majormoses]

I can certainly see the value in this but I think we need to be careful as to how this is introduced. While in the case of aws providers rate limits are more mitigated as most of their api limits are gated at the account level and rate limit awareness is built in but this is not true of all (or even most) providers. For example we use the github provider to manage teams, repos, etc. We have something like 30+ states/projects and if we ran them all at once it would most certainly fail.

[mwarkentin]

Could you make this opt in? atlantis apply --parallel

I'm dealing with about 50 RDS instances right now, each in their own projects - and need to enable multi-az, change instance type, etc. in different steps. This is incredibly slow waiting for each to happen 1 at a time.

My other alternative would be to make 50 separate PRs to "fake" the parallelization but that's tricky to manage as well.

[majormoses]

To better understand the use cases where this is important would it be something that makes sense as adhoc on PRs, repo config, server config, or yes? I am leaning towards it be adhoc on PRs as needed or repo config.

[mwarkentin]

I’m not too familiar with the distinction between repo config and server config, but it seems it would make sense to have a way to set the default behavior as well as enable or disable on an adhoc basis within a PR.

[darrylb-github]

👍 - was just about to ask about this. I have a repo with 4 projects and currently waiting on atlantis to finish planning as it's running them sequentially.

[Fauzyy]

We've been running a simple repo config implementation of parallel plans for a couple weeks in our fork and it's working well: https://github.com/runatlantis/atlantis/compare/v0.8.3..segmentio:v0.8.3-segment0.2.1

Caveat: because Atlantis has internal locks on the workspace name, this only works if all projects in the repo use Terraform Workspaces.

Example config:

version: 3
automerge: true
parallel_plans: true
...

This is for plans only, I feel that making applies parallel could have unintended side-effects.

There's also a flag to set the server-side cap on the # of concurrent goroutines that run per-request via --parallel-plans-pool-size=5 (default of 10)

Would love to get this or something similar into upstream

[YesYouKenSpace]

Will this be considered?

[lkysow]

Yeah that looks like a good solution since it avoids the problem of a lot of refactoring by only applying when using different workspaces. Segment's branch looks pretty diverged though so someone would need to make a clean PR.

[psalaberria002]

Does that mean we won't be able to parallelize it when not using workspaces?

[lkysow]

Does that mean we won't be able to parallelize it when not using workspaces?

Yes. But it's a first step. For not using workspaces the locking needs to be figured out so we don't modify a file-system that's got mutations in progress.

[Lowess]

@lkysow quick question as #926 was recently released under 0.13.0.

I am running Atlantis with a custom Terragrunt integration (really similar to the docs example

I wanted to leverage the newly introduced parallel features which is unfortunately not supported for Terragrunt workflows (yet) due to the workspace limitation.

I am wondering if I could fool atlantis by setting different workspaces (even though in terragrunt you don't need them) to make atlantis run plans and applies in parallel for prod / stage configs.

Is this a bad idea or you guys think that could be a valid workaround ? Looking at the plan output from terragrunt it looks good to me but I was wondering if there is something else I should worry about ?

• Here is my Terragrunt layout:

.
├── modules
└── terragrunt
    ├── account-1
    │   ├── account.hcl
    │   └── virginia
    │       ├── prod
    │       ├── region.hcl
    │       └── stage
    └── terragrunt.hcl

• Here is a sample from my atlantis.yaml:

---
version: 3

parallel_plan: true
parallel_apply: true
automerge: true

projects:
  # Project Anchor
  - &terragrunt
    name: template
    dir: '.null'
    workflow: terragrunt
    autoplan:
      enabled: true
      when_modified:
        - "./terraform/modules/**/*.tf"
        - "**/*.tf"
        - "**/terragrunt.hcl"


  - <<: *terragrunt
    name: account-1-virginia-prod-app-A
    dir: ./terraform/terragrunt/account-1/virginia/prod/app-A
    workspace: prod

  - <<: *terragrunt
    name: account-1-virginia-stage-app-A
    dir: ./terraform/terragrunt/account-1/virginia/stage/app-A
    workspace: stage

  [....]

Thanks for your feedback !

[lkysow]

Hi @Lowess, the reason this only works for workspaces is because Atlantis clones the repo into a separate directory for each workspace so there is no contention. If you set different workspaces then Atlantis will clone into separate directories so it should work.

[angeloskaltsikis]

@Lowess i wonder whether you have tried using Terragrunt with Terraform workspaces and how this experiment went? We are having multiple Terragrunt files and looking for ways on how we could make the {plan,apply} commands run on parallel as it may take atlantis a couple of hours to run a single command on 100s of Terragrunt leafs.

[ghostsquad]

Having the same problem, in that I have multiple projects in the same repo, all using different backends, etc. I'm not using any "workspaces" (Terraform definition of such), so everything is the "default" workspace. Parallel plans/apply don't work because it thinks that there's a shared lock in place. The first of 6 projects runs successfully, and the rest are reported that they cannot run because they are locked.

2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Running plans in parallel
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-1/us-west-2/prod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-2/us-west-2/nonprod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-3/us-west-2/prod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-3/us-west-2/nonprod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-1/us-west-2/nonprod/default"
2020/10/12 22:57:04+0000 [INFO] myorg/terraform-stuff#43: Acquired lock with id "myorg/terraform-stuff/deploy/environments/account-3/us-west-2/tools/default"
2020/10/12 22:57:06+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "cd \"${PWD%\"${REPO_REL_DIR}\"}\" && jb install\n" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 22:57:06+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "jsonnet -J \"${PWD%\"${REPO_REL_DIR}\"}/vendor\" terrasonnet.tf.jsonnet > terrasonnet.tf.json\n" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 22:57:10+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "/atlantis/data/bin/terraform0.13.4 init -input=false -no-color -upgrade" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 22:57:10+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "/atlantis/data/bin/terraform0.13.4 workspace show" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 22:57:13+0000 [EROR] myorg/terraform-stuff#43: Running "/atlantis/data/bin/terraform0.13.4 plan -input=false -refresh -no-color -out \"/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod/account-1-prod-us-west-2-default.tfplan\"" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod": exit status 1
2020/10/12 22:58:14+0000 [INFO] myorg/terraform-stuff#43: Successfully ran "/atlantis/data/bin/terraform0.13.4 plan -input=false -refresh -no-color" in "/atlantis/data/repos/myorg/terraform-stuff/43/default/deploy/environments/account-1/us-west-2/prod"
2020/10/12 23:00:30+0000 [INFO] server: Parsed comment as command="unlock" verbose=false dir="" workspace="" project="" flags=""

[richstokes]

+1 it would be awesome if we could plan in parallel multiple projects, assuming they are all on different backends.

[Sebor]

I've faced with the same issue. I use terragrunt without workspaces and I'd like to plan\apply in parallel. Does anybody know a workaround?

[mwarkentin]

Finally got a chance to try this out as we added a subset of projects which use Terraform Workspaces. It looks like I missed this caveat though:

Caveat: because Atlantis has internal locks on the workspace name, this only works if all projects in the repo use Terraform Workspaces.

I had assumed that it would allow us to plan the subset in parallel, and other projects would continue to function as before, however when we opened a PR with multiple (standard) projects each using the default workspace, ended up with The default workspace is currently locked by another command that is running for this pull request. Wait until the previous command is complete and try again. for all but one of the plans.

Would it be possible to support this use case? Maybe by supporting the parallel_[plan|apply] configuration at the project level instead of for the top level atlantis config?

[oliverisaac]

If we enable parallel_plan, it would be good to have a configuration item for how many parallel to run. Even a 2x parallel would be a huge relief for us

[ImIOImI]

A plan for us takes upwards of 30 minutes to execute due to having so many different environments. This isn't conducive to a healthy CI/CD where we want to fail fast.

[virgofx]

This would be an excellent feature to add. Similar to others, when Atlantis is managing multiple separate roots with different backends the serial nature increases execution time tremendously. Even if it were an opt-in feature such that terraform roots/projects could add some sort of -parallel feature such that it's still 100% backwards compatible would be a great addition.

[mihaiturcu]

any news here? :)

[Dawnflash]

This would be great, in our use case all projects differ in TF state and can absolutely run in parallel. This could be a project-level opt-out (parallel: false). It would save us lots of time.

[hussainbani]

Do we have any news on this? We want this feature.

[voidlily]

i think this was fixed in #2131, i've been able to use parallel by setting parallel_plan: true and parallel_apply: true on my repos

[nwsparks]

@voidlily it was rolled back. https://github.com/runatlantis/atlantis/issues/1205

it's only available if youre on the pre-release where it was present.

[voidlily]

Did they retract 0.19.3? That's the version I'm running and it works for me

[nitrocode]

Isn't this available now via https://www.runatlantis.io/docs/repo-level-atlantis-yaml.html#run-plans-and-applies-in-parallel ?

# atlantis.yaml in repo
parallel_plan: true
parallel_apply: true

cc: @voidlily @nwsparks @hussainbani @Dawnflash @jamengual @lkysow

I would prefer enabling this in the server configuration instead of in each terraform repo but a repo level config for this is better than serial plans and serial applies.

[jamengual]

I can't recall now but there was an issue and a PR that was reverted to allow a parallel plan per dir in each project, which is an issue for repos with many folders per env etc

[jamengual]

if someone has cycles: https://github.com/runatlantis/atlantis/pull/2253

The code is there, the issue is there and the suggested solution is documented......I mean......you can't have a better head start