Set custom location of '${HOME}/.terraform.d' directory for either Checkpoint client or Terraform CLI

[kmezynski]

Terraform Version

Terraform v1.5.5
on linux_amd64

Use Cases

To enhance flexibility and maintain a clean $HOME directory, it would be beneficial to have the ability to set a custom directory for either the Checkpoint service or the Terraform CLI configuration directory (not the file) through an environmental variable, similar to the approach used for TF_CLI_CONFIG_FILE. 😉

Attempted Solutions

Please correct me if I'm wrong, but configuring Terraform environmental variables for CLI config does not affect the location where the Checkpoint client generates its files during the execution of the terraform init command. These files are consistently generated in the following hard coded locations:

1. ${HOME}/.terraform.d/checkpoint_cache
2. ${HOME}/.terraform.d/checkpoint_signature

Exemplary configuration attempt:

export XDG_CONFIG_HOME="${HOME}/.config"
export XDG_DATA_HOME="${HOME}/.local/share"
export TF_CLI_CONFIG_FILE="${XDG_CONFIG_HOME}/terraform/terraformrc"
export TF_PLUGIN_CACHE_DIR="${XDG_DATA_HOME}"

Proposal

1. Implement the feature to terraform CLI Overriding of Checkpoint configuration through environmental variables, similar to what is done in the LoadConfig function that calls the cliConfigFileOverride function, I suggest the use of the name TF_CLI_CONFIG_DIR. It's possible that this could eventually replace TF_CLI_CONFIG_FILE, but I am not fully aware of all the consequences of such a change, except for some obvious considerations such as maintaining CLI backward compatibility. Alternatively, a more specific name related solely to the Checkpoint service could also be considered (point 2.)

2. Implement the feature to go-checkpoint client Since the Checkpoint client is not exclusive to Terraform but is also used in other products like Packer, Vagrant, and more, it might be more appropriate to propose a similar improvement as mentioned in the previous point directly to the go-checkpoint repository.

I would be enthusiastic about implementing such an improvement, but I would appreciate input from the community to determine which approach is more reasonable (if any).

References

No response

[apparentlymart]

See also: #15389

[kmezynski]

@apparentlymart thank you for the reference, I may have overlooked it initially. After thorough research encompassing related issues and closed PRs, it appears that there were several primary challenges that impeded the implementation of this particular feature:

1. A lower priority in comparison to other features being developed concurrently
2. Varying conventions across different Unix-like operating systems, with XDG base directories not being widely adopted on macOS
3. The need to maintain backward compatibility was a significant concern
4. Cache-ish and config-ish nature of files that are being stored in ${HOME}/.terraform.d

However, I have not come across any evidence of substantial progress towards enabling a custom path for ${HOME}/.terraform.d beyond extensive discussions. Do you have any suggestions on how we can advance this topic?

[apparentlymart]

Hi @kmezynski! Sorry for the terseness of my previous reply. I was intending that as a note to others on our team for context from the previous discussion, not as criticism that you didn't find the other issue.

[ascopes]

This would probably be beneficial to builds that run on platforms like Jenkins where multiple jobs may run on the same node and thus inherit the same environment concurrently.

[next-jesusmanuelnavarro]

Another corner case in favour of this feature: as of now I'm working on a docker container for terraform operations and using a private Artifactory remote state. As such, I'm injecting a ~/.terraform.d/credentials.trfc.json file as a mounted volume in docker compose, like this:

(...)
    volumes:
     - ${PWD}/secrets/credentials.tfrc.json:/home/operator/.terraform.d/credentials.tfrc.json

Unluckily, this means the directory /home/operator/.terraform.d is owned by root:root and the non-privileged user that runs terraform can't write /home/operator/.terraform.d/checkpoint_cache nor /home/operator/.terraform.d/checkpoint_signature files and so, terraform plan, terraform apply fail.

Any of these could solve this situation:

• honor ~/.netrc contents and $NETRC environment variable.
• let credentials.trfc.json be on a different path exposed by an environment variable.
• set the writeable directory for those checkpoints to be configurable.

Yes, I know I can use the TF_TOKEN_[server] environment variable but this also poses in my case different problems out of the scope of this conversation.