Using --preserve-projects with --filter pointing to a directory which already has representation leads to a duplicate

[gmaghera]

Hello,

Thanks again for implementing incremental builds. I'm integrating it into our workflow and there might be an edge case bug.

For example, I have a TG module at path tools/us-east-1/debug. If I run the following I end up with a second stanza for the same.

❯ terragrunt-atlantis-config generate --ignore-parent-terragrunt --autoplan --create-workspace --parallel --automerge --filter tools/us-east-1/debug --preserve-projects --output atlantis.yaml; grep -F "tools/us-east-1/debug" atlantis.yaml
INFO[0000] Created project for /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-1/debug/terragrunt.hcl
  dir: tools/us-east-1/debug
  dir: tools/us-east-1/debug

Would it be possible to replace instead?

Cheers, Gabor

[ax-nmac]

We noticed the same problem. I got around this using yq which is like jq for yaml, but its hacky at best. Would love for this to be fixed on the terragrunt-atlantis-config level if possible.

[gmaghera]

That is exactly what we are doing too, @ax-nmac.

Our Terragrunt repository is so large (over 6000 TG modules) that it takes 20 minutes to run a full generation.

Using partial generation reduces that to 3 seconds, for a single Terragrunt module! But then the yq handling adds another 9 seconds, so this workaround adds a 300% performance penalty.

[dmattia]

We could likely update this by updating this line: https://github.com/transcend-io/terragrunt-atlantis-config/blob/master/cmd/generate.go#L751 to remove existing projects with duplicate keys when appending the new ones.

[gmaghera]

@dmattia can you assign this issue to me?

[gmaghera]

Thank you for reviewing my change, @dmattia!

Below are the performance gains we're getting, in our script wrapping terragrunt-atlantis-config. From ~27 seconds to ~2 seconds!

Before fix, using a workaround:

❯ time ./scripts/atlantis-generate.sh tools/us-east-2/debug/terragrunt.hcl
✅ Found version 1.15.0 of terragrunt-atlantis-config installed.
✅ Found version 3.1.0 of yq(python-yq) installed.
INFO[0001] Created project for /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/debug/terragrunt.hcl
./scripts/atlantis-generate.sh tools/us-east-2/debug/terragrunt.hcl  24.53s user 1.53s system 95% cpu 27.188 total

After fix:

❯ time ./scripts/atlantis-generate.sh tools/us-east-2/debug/terragrunt.hcl
✅ Found version 1.16.0 of terragrunt-atlantis-config installed.
INFO[0001] Updated project for /Users/gmaghera/Workspaces/dc-terraform/tools/us-east-2/debug/terragrunt.hcl
./scripts/atlantis-generate.sh tools/us-east-2/debug/terragrunt.hcl  1.67s user 0.35s system 101% cpu 2.003 total

[stdmje]

Hi @gmaghera! Could you please explain how to work with incremental builds? I am facing the same problem where our number of TG projects are growing and the time for generate the atlantis.yaml file is taking some minutes now.

Here is current repo config.

repoConfig: |
  ---
  repos:
  - id: /.*/
    delete_source_branch_on_merge: false
    apply_requirements: [mergeable, approved]
    allowed_workflows: [terragrunt]
    workflow: terragrunt
    pre_workflow_hooks:
    - run: terragrunt-atlantis-config generate --filter "terragrunt/clients/[a-z]*" --output atlantis.yaml --autoplan=false --parallel=true --num-executors=5 --create-workspace --execution-order-groups --cascade-dependencies=false
  workflows:
    terragrunt:
      plan:
        steps:
        - run: terragrunt providers lock -platform=linux_amd64
        - run: export TF_PLUGIN_CACHE_DIR="/atlantis-data/plugin-cache" && terragrunt run-all plan -no-color -lock=false --terragrunt-ignore-external-dependencies -out=$PLANFILE
      apply:
        steps:
        - run: export TF_PLUGIN_CACHE_DIR="/atlantis-data/plugin-cache" && terragrunt run-all apply -no-color --terragrunt-non-interactive --terragrunt-ignore-external-dependencies

Thanks in advance!!