Issues while using path_relative_to_include in backend config remote state

[m8051]

Howdy folks.

I am prototyping some IaC and wanted to try out Terragrunt, but I am struggling to run some of the built-in functions, concretely thepath_relative_to_include() which returns . instead of the relative path between the current terragrunt.hcl file and the path specified in my include block.

Using terragrunt version v0.38.10 / terraform v1.2.9

Here's my layout:

.
├── README.md
├── backend.tf
├── environments
│   ├── development
│   │   └── ec2-instance
│   │       └── terragrunt.hcl
│   ├── production
│   │   └── ec2-instance
│   │       └── terragrunt.hcl
│   └── staging
│       └── ec2-instance
│           └── terragrunt.hcl
├── modules
│   └── ec2-instance
│       ├── main.tf
│       ├── outputs.tf
│       ├── provider.tf
│       └── variables.tf
└── terragrunt.hcl

9 directories, 10 files

The root terragrunt.hcl

locals {
  env_name = replace(path_relative_to_include(), "environments/", "")
}

inputs = {
  env_name= local.env_name
}

# Configure S3 and DynamoDB
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket         = "environment-${local.env_name}"
    key            = "${local.env_name}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-statefile-locks-${local.env_name}"
  }
}

The development/ec2-instance/terragrunt.hcl

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "../../../modules/ec2-instance"
}

inputs = {
  instance_type = "t2.micro"
  instance_name = "testing-terragrunt-development"
  instance_ami  = "ami-05fa00d4c63e32376"
  vpc_security_group_ids = ["sg-xxxxxxxx"]
  subnet_id = "subnet-xxxxxx"
}

The error when trying to provision the s3 bucket

terragrunt validate
Remote state S3 bucket environment-. does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) n
ERRO[0002] remote state S3 bucket environment-. does not exist or you don't have permissions to access it 
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 

And the backend.tf file autogenerated:

terraform {
  backend "s3" {
    bucket         = "environment-."
    dynamodb_table = "terraform-statefile-locks-."
    encrypt        = true
    key            = "./terraform.tfstate"
    region         = "us-east-1"
  }

Am I missing something?

Thanks in advance.

[denis256]

Hello, terragrunt validate is invoked in which directory? in development/ec2-instance ? or in the root of the project?

In my tests, it works fine if invoked in development/ec2-instance directory

Reference: https://github.com/denis256/terragrunt-tests/tree/master/issue-2277/development/ec2-instance

[m8051]

Hi @denis256 , thanks for checking this out for me, I run terragrunt from the root directory as I'd like to provision the 3 ec2s based on environments at once.

Is there a better way to achieve this behavior? I might have misunderstood the main idea perhaps?

[denis256]

Hi, most probably invoking run-all will help

https://terragrunt.gruntwork.io/docs/features/execute-terraform-commands-on-multiple-modules-at-once/#the-run-all-command

[m8051]

Hi, most probably invoking run-all will help

https://terragrunt.gruntwork.io/docs/features/execute-terraform-commands-on-multiple-modules-at-once/#the-run-all-command

@denis256 Are you suggesting that in order to execute terragrunt -run-all in the root directory I have to previously run terragrunt validate|init individually for each environment first ?

Or simply by running terragrunt -run-all with nothing initialized would be sufficient?

Thanks again for your help

[m8051]

Ok I managed to make some progress on that but still I have something misconfigured on my end:

This is the output when running terragrunt run-all plan -out=all.plan

prototype git:(layout) ✗ terragrunt run-all plan -out=all.plan
INFO[0000] The stack at /Users/xx/repos/prototype will be processed in the following order for command plan:
Group 1
- Module /Users/xx/repos/prototype/environments/development/ec2-instance
- Module /Users/xx/repos/prototype/environments/production/ec2-instance
- Module /Users/xx/repos/prototype/environments/staging/ec2-instance

Remote state S3 bucket staging/ec2-instance does not exist or you don't have permissions to access it. 
Would you like Terragrunt to create it? (y/n)

Remote state S3 bucket development/ec2-instance does not exist or you don't have permissions to access it. 
Would you like Terragrunt to create it? (y/n) 

Remote state S3 bucket production/ec2-instance does not exist or you don't have permissions to access it.
Would you like Terragrunt to create it? (y/n)

And what I'd like is to have the s3 buckets to contain only the name of the environment, for example:

Remote state S3 bucket environments-staging
Remote state S3 bucket environments-development
Remote state S3 bucket environments-production

Any tips on how to make those changes?

[denis256]

Hi, I was thinking to split path_relative_to_include path by "/" and take the first part

env_name = split("/", path_relative_to_include())[0]

[m8051]

@denis256 sorry for the delay, I tried by splitting the path_relative_to_include and taking the first part but does not work either

These are my config files:

locals {
  # fetch_environments = replace(path_relative_to_include(), "environments/", "")
  # environments       = replace(local.fetch_environments, "/ec2-instance", "")
  environments = split("/", path_relative_to_include())[0]
}

inputs = {
  environments= local.environments
}

# Configure S3 and DynamoDB
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket         = "${local.environments}"
    key            = "${local.environments}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-statefile-locks-${local.environments}"
  }
}

.
├── backend.tf
├── environments
│   ├── development
│   │   └── ec2-instance
│   │       └── terragrunt.hcl
│   ├── production
│   │   └── ec2-instance
│   │       └── terragrunt.hcl
│   └── staging
│       └── ec2-instance
│           └── terragrunt.hcl
├── modules
│   └── ec2-instance
│       ├── main.tf
│       ├── outputs.tf
│       ├── provider.tf
│       └── variables.tf
└── terragrunt.hcl

When running Terragrunt from the root directory:

➜  terragrunt git:(testing-terragrunt) ✗ terragrunt run-all init
INFO[0000] The stack at xxxx/yyyy terragrunt will be processed in the following order for command init:
Group 1
- Module xxxx/yyyy/terragrunt
- Module xxxx/yyyy/terragrunt/environments/development/ec2-instance
- Module xxxx/yyyy/terragrunt/environments/production/ec2-instance
- Module xxxx/yyyy/terragrunt/environments/staging/ec2-instance

Remote state S3 bucket . does not exist or you don't have permissions to access it. 
Would you like Terragrunt to create it? (y/n) 
Remote state S3 bucket environments does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) 
Remote state S3 bucket environments does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) 
Remote state S3 bucket environments does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) ^C

Any other thing that I could try?

[denis256]

Hi, setting environments = split("/", path_relative_to_include())[1] is not helping?

[m8051]

@denis256 does not help ... this is the output when setting the environments = split("/", path_relative_to_include())[1]

terragrunt git:(testing-terragrunt) ✗ terragrunt init
ERRO[0000] Error: Invalid index
               
ERRO[0000]   on xxxx/yyyy/terragrunt/terragrunt.hcl line 4, in locals: 
ERRO[0000]    4:   environments = split("/", path_relative_to_include())[1] 
ERRO[0000]                                              
ERRO[0000] The given key does not identify an element in this collection value.
 
ERRO[0000] Encountered error while evaluating locals in filexxxx/yyyy/terragrunt/terragrunt.hcl
ERRO[0000] xxxx/yyyy/terragrunt/terragrunt.hcl/terragrunt/terragrunt.hcl:4,56-59: Invalid index; The given key does not identify an element in this collection value. 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[denis256]

Hi, not sure if is required to do init in the repository root, also init is not required at all because of auto-init function in terragrunt

I think should be executed only run-all

terragrunt run-all <plan/apply/destroy>

[m8051]

Hi,

not sure if is required to do init in the repository root, also init is not required at all because of auto-init function in terragrunt

I think should be executed only run-all

terragrunt run-all <plan/apply/destroy>

Cheers @denis256 for all your help on this one, Happy to close this out