Add function which return git folder path for backend configuration

[adv4000]

OpenTofu Version

N/A

Use Cases

When you configure remote state with AWS S3 we need to set Key which is basically path in S3 bucket to store remote state, Usually this is equal path in git repository For example in TF:

tofu {
  backend "s3" {
    bucket = "adv-it-terraform-remote-state" // Bucket where to SAVE Terraform State
    key    = "dev/network/terraform.tfstate" // Object name in the bucket to SAVE Terraform State
    region = "us-west-2"                     // Region where bucket created
  }
}

dev/network/ represent path in my git folder. Pease add Tofu function which will return Git path from the current folder, basically execute this command on the background: git rev-parse --show-prefix

So the solution will be something like this:

tofu {
  backend "s3" {
    bucket = "adv-it-terraform-remote-state" // Bucket where to SAVE Terraform State
    key    = "${gitpath}/terraform.tfstate"  // Object name in the bucket to SAVE Terraform State
    region = "us-west-2"                     // Region where bucket created
  }
}

Attempted Solutions

We using makefile or other wrapper for this.

Proposal

Add Tofu function gitpath to return folder path from current folder to root folder of git repository.

References

N/A

[maunzCache]

You might want to consider terragrunt as short term solution. See https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#get_path_to_repo_root and https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#find_in_parent_folders .

[adv4000]

We using makefile or other wrapper for this.

We using makefile or other wrapper for this like terragrunt already.

[Hefeweizen]

You note you have a work-around already. I'm not sure what that is, so I'll say: if you have GITPATH as an environment variable, then key = "{{ GITPATH }}/terraform.tfstate" would be interpolated.

However, neither functions nor variables can be used in these blocks. So, there's a larger issue, and there's an issue for it: https://github.com/opentofu/opentofu/issues/388

If it were possible to use a function/variable here, gitpath() seems very narrow. Would you be happy with a system() or exec() function instead? You could then do something like:

locals {
  gitpath = system("git rev-parse HEAD")
}

tofu {
  backend "s3" {
    bucket = "adv-it-terraform-remote-state" // Bucket where to SAVE Terraform State
    key    = "${local.gitpath}/terraform.tfstate"  // Object name in the bucket to SAVE Terraform State
    region = "us-west-2"                     // Region where bucket created
  }
}

[adv4000]

You note you have a work-around already. I'm not sure what that is, so I'll say: if you have GITPATH as an environment variable, then key = "{{ GITPATH }}/terraform.tfstate" would be interpolated.

However, neither functions nor variables can be used in these blocks. So, there's a larger issue, and there's an issue for it: #388

If it were possible to use a function/variable here, gitpath() seems very narrow. Would you be happy with a system() or exec() function instead? You could then do something like:

locals {
  gitpath = system("git rev-parse HEAD")
}

tofu {
  backend "s3" {
    bucket = "adv-it-terraform-remote-state" // Bucket where to SAVE Terraform State
    key    = "${local.gitpath}/terraform.tfstate"  // Object name in the bucket to SAVE Terraform State
    region = "us-west-2"                     // Region where bucket created
  }
}

Hi, Yes Function like system() will be great!

[kislerdm]

@adv4000 hey Denis! Thank you for your proposal. Could you please share some more details about the use case? Did I get it right that it address facilitation of multi-state management with mono-repo?

Pease add Tofu function which will return Git path from the current folder, basically execute this command on the background

IMO explicit propagation of input vars to the module is a cleaner approach compared to tofu invoking external process implicitly.

[adv4000]

@adv4000 hey Denis! Thank you for your proposal. Could you please share some more details about the use case? Did I get it right that it address facilitation of multi-state management with mono-repo?

Pease add Tofu function which will return Git path from the current folder, basically execute this command on the background

IMO explicit propagation of input vars to the module is a cleaner approach compared to tofu invoking external process implicitly.

Hi @kislerdm , Yes it's automated dynamic way to set different path in S3 bucket for remote state. When we deploy infrastructure using TF, it never from one folder! We have many folders, For example:

    ├── dev
    │       ├── s3
    │       └── vpc
    │            ├── applications
    │            │   ├── app1
    │            │   └── app2
    │            ├── databases
    │            └── vpn
    └─── prod
              ├── s3
              └── vpc
                  ├── applications
                  │   ├── app1
                  │   └── app2
                  ├── databases
                  └── vpn

Deployment from Each folder has it's own path in s3 bucket for remote state, in every folder we need to hardcode this path. usually this path equal to output of git rev-parse HEAD command which return path from .git folder to current location. If this can be dynamic, no need to hardcode this anymore, people using terragrunt just because of this limitation of TF in additional to any reasons.

[kislerdm]

Hey @adv4000! Thanks for the illustration and details.

May I kindly ask you to clarify the following question to help me understand the requirements better?

1. How states are defined using the configurations stored according to the project structure you shared? In other words, how many states are there on the illustration: two - dev and prod, four - {prod, dev}/{s3, vpc}, eight - {prod, dev}/{s3, vpc/applications, vpc/databases, vpc/vpn}, or ten - {prod, dev}/{s3, vpc/applications/app1, vpc/applications/app2, vpc/databases, vpc/vpn}?
2. What is the reason for coupling the states location in the backend to the directories structure? For context, let's imaging, your team decides to restructure the repo and moves the tofu configuration from the root directory, i.e. the prod and dev dirs will be moved to the directory product1, would you want to migrate the states in the backed accordingly?

My personal opinion. I find execution of a subprocess from tofu to configure backend prone to side effects, and I'd rather suggest to avoid it. It feels like misuse of OpenTofu, because it becomes template generator relying on extra configuration to generate OpenTofu configuration. IMO tofu backend configuration should not be "aware" of its location on disk, and tofu modules should not depend on the content of parent directories.

About introduction of a function to make system calls. terraform_data resource can be used to execute subprocesses if it's needed.

Personally, I'm not a proponent of the idea of introducing such function like system, but it'd be crucial to collect more opinions shared by the community to take any decision.

Another point: we may consider tofu functionality extension to generate OpenTofu configuration using templates. For example, it can be something similar to how kustomize became part of kubectl.

WDYT?

[adv4000]

Hey @adv4000! Thanks for the illustration and details.

May I kindly ask you to clarify the following question to help me understand the requirements better?

1. How states are defined using the configurations stored according to the project structure you shared? In other words, how many states are there on the illustration: two - dev and prod, four - {prod, dev}/{s3, vpc}, eight - {prod, dev}/{s3, vpc/applications, vpc/databases, vpc/vpn}, or ten - {prod, dev}/{s3, vpc/applications/app1, vpc/applications/app2, vpc/databases, vpc/vpn}?
2. What is the reason for coupling the states location in the backend to the directories structure? For context, let's imaging, your team decides to restructure the repo and moves the tofu configuration from the root directory, i.e. the prod and dev dirs will be moved to the directory product1, would you want to migrate the states in the backed accordingly?

My personal opinion. I find execution of a subprocess from tofu to configure backend prone to side effects, and I'd rather suggest to avoid it. It feels like misuse of OpenTofu, because it becomes template generator relying on extra configuration to generate OpenTofu configuration. IMO tofu backend configuration should not be "aware" of its location on disk, and tofu modules should not depend on the content of parent directories.

About introduction of a function to make system calls. terraform_data resource can be used to execute subprocesses if it's needed.

Personally, I'm not a proponent of the idea of introducing such function like system, but it'd be crucial to collect more opinions shared by the community to take any decision.

Another point: we may consider tofu functionality extension to generate OpenTofu configuration using templates. For example, it can be something similar to how kustomize became part of kubectl.

WDYT?

1. Ten, each deployment folder have it;s own remote state.
2. This is in my opinion is the best approach, each deployment layer has it's own directory and it's own remote state

Terragrunt used by many people just because of this TF limitation.