Terraagrunt Run-all is not processing external dependencies correctly

[laughtonsm]

I am experiencing issues whilst trying to deploy a "Stack" using the run-all functionality Details are as follows:

Terraform Version: 0.15.5 Terragrunt Version: 0.31.0

Behaviour:

Currently have external dependencies which are outside the current working directory. When running terragrunt run-all plan, the path of the external dependency is correctly identified. However, when proceeding with the plan, the current working director path is used, and the plan fails as the dependency is not found. See below:

root@95ac24363805:/tg/test/pltfrm-tst-connectivity-001/uksouth# terragrunt run-all plan -terragrunt-include-external-dependencies
Module /tg/test/pltfrm-tst-connectivity-001/uksouth/key_vault_mgmt_hosts depends on module /tg/test/pltfrm-tst-management-001/uksouth/svc_central_logging, which is an external dependency outside of the current working directory. Should Terragrunt run this external dependency? Warning, if you say 'yes', Terragrunt will make changes in /tg/test/pltfrm-tst-management-001/uksouth/svc_central_logging as well! (y/n) y
ERRO[0011] Error processing module at '/tg/test/pltfrm-tst-connectivity-001/uksouth/svc_central_logging/terragrunt.hcl'. How this module was found: dependency of module at '/tg/test/pltfrm-tst-connectivity-001/uksouth/resource_group_mgmt_hosts'. Underlying error: Error reading file at path /tg/test/pltfrm-tst-connectivity-001/uksouth/svc_central_logging/terragrunt.hcl: open /tg/test/pltfrm-tst-connectivity-001/uksouth/svc_central_logging/terragrunt.hcl: no such file or directory
ERRO[0011] Unable to determine underlying exit code, so Terragrunt will exit with error code 1
root@95ac24363805:/tg/test/pltfrm-tst-connectivity-001/uksouth#

The external dependency has already been applied and has outputs

Wondering if this is actually expected behaviour or not

[brikis98]

Hm, this does seem like a bug. Could you share how the dependency is defined in your terragrunt.hcl?

[azec-pdx]

I am experiencing same behavior on init and validate subcommands of the terragrunt run-all command. Debugging shows me that -–terragrunt-include-external-dependencies flag is consumed properly by Terragrunt, but as a buggy side-effect it also passes this as a flag of corresponding/underlying terraform command.

Some examples...

When I run:

TF_LOG=DEBUG terragrunt run-all init --terragrunt-log-level debug --terragrunt-debug --terragrunt-non-interactive -–terragrunt-include-external-dependencies

one of the debug lines shows this:

time=2021-09-14T15:46:11-07:00 level=debug msg=Running command: terraform init -backend-config=bucket=t<REDACTED> -backend-config=dynamodb_table=terraform-terragrunt-locks -backend-config=encrypt=true -backend-config=key=rds_faadb11/terraform.tfstate -backend-config=region=us-west-2 -backend-config=role_arn=arn:aws:iam::<REDACTED>:role/<REDACTED> -input=false -–terragrunt-include-external-dependencies prefix=[/Users/amer/dev/github/skywardio/infra-tf/infra/nonprod/rds_faadb11]

which being distilled to just terraform command is:

terraform init -backend-config=bucket=<REDACTED> -backend-config=dynamodb_table=terraform-terragrunt-locks -backend-config=encrypt=true -backend-config=key=rds_faadb11/terraform.tfstate -backend-config=region=us-west-2 -backend-config=role_arn=arn:aws:iam::<REDACTED>:role/<REDACTED> -input=false -–terragrunt-include-external-dependencies

---

$ terragrunt --version
terragrunt version v0.28.16

$ terraform --version
Terraform v0.14.11
+ provider registry.terraform.io/hashicorp/archive v1.3.0
+ provider registry.terraform.io/hashicorp/aws v3.53.0
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/tls v3.1.0
+ provider registry.terraform.io/newrelic/newrelic v2.25.0

Your version of Terraform is out of date! The latest version
is 1.0.6. You can update by downloading from https://www.terraform.io/downloads.html

Directory structure: infra/_global/newrelic/terragrunt.hcl infra/nonprod/rds_faadb11/terragrunt.hcl -> WORKING DIR (from which above cmd was issued) infra/nonprod/central_logs/terragrunt.hcl infra/nonprod/eks_preprod/vpc_preprod_k8s/terragrunt.hcl infra/nonprod/eks-1.21/core/terragrunt.hcl infra/nonprod/vpc_general/terragrunt.hcl

Content of infra/nonprod/rds_faadb11/terragrunt.hcl:

terraform {
  source = "${get_parent_terragrunt_dir()}/../modules//profiles/skyward_rds"
}

include {
  path = find_in_parent_folders()
}

dependencies {
  paths = [
    "../central_logs",
    "../eks_preprod/vpc_preprod_k8s", # Clean up after decommissioning preprod 1.16 cluster
    "../eks-1.21/core",
    "../../_global/newrelic",
    "../vpc_general",
  ]
}

dependency "central_logs" {
  config_path = "../central_logs"

  mock_outputs = {
    bucket_id_access_s3 = "mock-bucket-id"
  }
}

dependency "k8svpc" {
  config_path = "../eks_preprod/vpc_preprod_k8s"

  mock_outputs = {
    name        = "mock"
    namespace   = "mock"
    environment = "mock"
    region      = "us-east-1"

    vpc_ipv4_cidr = ["127.0.0.0/16"]
  }
}

dependency "eks_core" {
  config_path = "../eks-1.21/core"

  mock_outputs = {
    vpc = {
      vpc_ipv4_cidr = "10.0.0.0/16"
    }
  }
  mock_outputs_allowed_terraform_commands = ["init", "validate"]
}

dependency "newrelic" {
  config_path = "../../_global/newrelic"
}

dependency "vpc" {
  config_path = "../vpc_general"
}

inputs = {
  ...
  //REDACTED
  ...
}

[azec-pdx]

@brikis98 ☝️ , I am hoping this helps with better understanding of the problem ...

[azec-pdx]

Long story short, this prevents commands like terragrunt run-all validate with auto-init to run in CI/CD flows without human interaction to prompts and wanting to do init on all external dependencies. In our use-case, we ended up having new AWS EKS cluster as a new dependency to the module that pre-existed for long time. We don't have control of ordering of directories in CI/CD and parallelism involved to control which modules get init before others. Relying on terragrunt run-all validate --terragrunt-non-interactive -–terragrunt-include-external-dependencies was our best shot.

[dzavalkin-scayle]

@brikis98 Any chance this bug is finally fixed and/or these flags are removed from the docs https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-ignore-external-dependencies ? What is the point of these flags if terraform command invoked by terragrunt will fail if we provide -–terragrunt-include-external-dependencies flag to terragrunt command?

[yiskaneto]

I ran into this today while using the Terragrunt action to spin an EKS cluster, to address this (as mentioned by @azec-pdx) I added the --terragrunt-include-external-dependencies flag to the command passed to the Terragrunt action input , since the same command is used on different jobs I defined it as a global workflow environment:

env:
  TG_COMMAND: 'run-all ${{ inputs.tg-command}} --terragrunt-include-external-dependencies'

So more than a bug, in my case was a matter of using the correct flag on the ci/cd pipeline.