terraform icon indicating copy to clipboard operation
terraform copied to clipboard

Published 20 hours ago • verified publisher icon hashicorp

Backend Config file cannot be read following an (unnecessary?) absolute/relative path transformation

Open wigbam opened this issue 2 years ago • 1 comments

Terraform Version

Terraform v1.3.3
on linux_amd64

Terraform Configuration Files

terraform {
  required_version = "~> 1.0"

  backend "s3" {
    # to be provided externally
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.2.0"
    }
  }

}

Debug Output

skolesnik@Stepan-XPS:~/Documents/workspace/spike/category-data/terraform$ TF_LOG=trace terraform init -input=false -back
end-config="/mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars"
2022-10-24T19:51:06.331+0200 [INFO]  Terraform version: 1.3.3
2022-10-24T19:51:06.331+0200 [DEBUG] using github.com/hashicorp/go-tfe v1.9.0
2022-10-24T19:51:06.331+0200 [DEBUG] using github.com/hashicorp/hcl/v2 v2.14.1
2022-10-24T19:51:06.331+0200 [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2022-10-24T19:51:06.331+0200 [DEBUG] using github.com/hashicorp/terraform-svchost v0.0.0-20200729002733-f050f53b9734
2022-10-24T19:51:06.331+0200 [DEBUG] using github.com/zclconf/go-cty v1.11.1
2022-10-24T19:51:06.331+0200 [INFO]  Go runtime version: go1.19.1
2022-10-24T19:51:06.331+0200 [INFO]  CLI args: []string{"terraform", "init", "-input=false", "-backend-config=/mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars"}
2022-10-24T19:51:06.331+0200 [TRACE] Stdout is a terminal of width 120
2022-10-24T19:51:06.331+0200 [TRACE] Stderr is a terminal of width 120
2022-10-24T19:51:06.331+0200 [TRACE] Stdin is a terminal
2022-10-24T19:51:06.331+0200 [DEBUG] Attempting to open CLI config file: /home/skolesnik/.terraformrc
2022-10-24T19:51:06.331+0200 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2022-10-24T19:51:06.331+0200 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2022-10-24T19:51:06.331+0200 [DEBUG] ignoring non-existing provider search directory /home/skolesnik/.terraform.d/plugins
2022-10-24T19:51:06.331+0200 [DEBUG] ignoring non-existing provider search directory /home/skolesnik/.local/share/terraform/plugins
2022-10-24T19:51:06.331+0200 [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2022-10-24T19:51:06.331+0200 [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2022-10-24T19:51:06.331+0200 [DEBUG] ignoring non-existing provider search directory /var/lib/snapd/desktop/terraform/plugins
2022-10-24T19:51:06.332+0200 [INFO]  CLI command args: []string{"init", "-input=false", "-backend-config=/mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars"}

Initializing the backend...
╷
│ Error: Failed to read file
│
│ The file
│ "../../../../../../../mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars" could
│ not be read.
╵

Expected Behavior

The backend-config file should have been successfully loaded from the absolute path (without a transformation to a relative one) as the file itself is perfectly readable via the provided absolute path:

ls -la /mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvarsolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars
-rwxrwxrwx 1 skolesnik skolesnik 120 Oct 24 19:38 /mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars

Actual Behavior

The terraform failed to load the backend-config file.

Initializing the backend...
╷
│ Error: Failed to read file
│
│ The file
│ "../../../../../../../mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars" could
│ not be read.
╵

Steps to Reproduce

terraform init -input=false -backend-config="$TF_BACKEND_CONFIG_PATH"

There are indications that the issue could be caused by the WSL's strange handling of symlinks. See additional context for more information.

Additional Context

  1. Terraform is used from Ubuntu 20.04 LTS on top of WSL (Windows 11 PRO 22H2).
  2. The parent directory of where the terraform files live is symlinked to the user profile on the main Windows drive as per below:
skolesnik@Stepan-XPS:~$ ls -l
total 0
lrwxrwxrwx 1 skolesnik skolesnik 32 Oct 24 18:31 Documents -> /mnt/c/Users/skolesnik/Documents
lrwxrwxrwx 1 skolesnik skolesnik 32 Oct 24 18:31 Downloads -> /mnt/c/Users/skolesnik/Downloads
lrwxrwxrwx 1 skolesnik skolesnik 31 Oct 24 18:31 Pictures -> /mnt/c/Users/skolesnik/Pictures
skolesnik@Stepan-XPS:~$ readlink -f ~/Documents/workspace/spike/category-data/target/tf_backend.tfvars
/mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars
  1. Weirdly, the relative path which Terraform transforms the provided absolute path to seems to be correct, but it cannot be read by the file system either, see below:
skolesnik@Stepan-XPS:~/Documents/workspace/spike/category-data/terraform$ pwd
/home/skolesnik/Documents/workspace/spike/category-data/terraform
skolesnik@Stepan-XPS:~/Documents/workspace/spike/category-data/terraform$ terraform init -input=false -backend-config=/mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars

Initializing the backend...
╷
│ Error: Failed to read file
│
│ The file
│ "../../../../../../../mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars" could
│ not be read.
╵
skolesnik@Stepan-XPS:~/Documents/workspace/spike/category-data/terraform$ ls -la ../../../../../../../mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars
ls: cannot access '../../../../../../../mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars': No such file or directory
  1. Even more weirdly, if the relative path is slightly amended by prepending two additional ../ then the file can be accessed:
skolesnik@Stepan-XPS:~/Documents/workspace/spike/category-data/terraform$ ls -la ../../../../../../../../../mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars
-rwxrwxrwx 1 skolesnik skolesnik 120 Oct 24  2022 ../../../../../../../../../mnt/c/Users/skolesnik/Documents/workspace/spike/category-data/target/tf_backend.tfvars
  1. Regardless of all of the above, shouldn't Terraform just use the absolute path if it is explicitly provided by the user?
  2. Naturally, working from a non-symlinked directory works fine

References

  • #24196

wigbam avatar Oct 24 '22 19:10 wigbam

Thanks for reporting this, @wigbam.

In several different cases Terraform attempts to transform a provided path into a relative one (relative to the root module) so that it's safe to store the path in persistent locations such as in state snapshots without the paths churning when the module is run on different computers.

However, you're right that there's not really any reason for Terraform to be normalizing this particular path: it's only used temporarily by terraform init to load this file and not persisted anywhere once that command is complete. I think this is happening just because the code which loads the backend configuration file is sharing a common codepath used to load some other kinds of files where that normalization is necessary, and so it's over-zealously changing this path on the way through.

This hasn't come up before I suppose because folks generally don't use absolute paths to populate these settings; Terraform generally expects to be working within the confines of a particular prefix such as a version control work tree. But that doesn't mean we can't make it work for this particular case, assuming that we can make sure that skipping the relativization for this particular case doesn't break some other case where it's important.

Thanks again!

apparentlymart avatar Oct 26 '22 23:10 apparentlymart

I'm not sure if it's the same issue, but it's the closest that I've come to describing what we're experiencing.

When I run terraform plan on any version of terraform >= 1.3.0 (it doesn't occur on 1.2.9) on a directory with a symlinked modules/ directory, I get an error to the effect of:

Running plan in the remote backend. Output will stream here. Pressing Ctrl-C
will stop streaming the logs, but will not stop the plan running remotely.

Preparing the remote plan...

Error: Failed to upload configuration files: failed to get relative path for file "/Users/eddiebachle/git/stockx/data-ai-platform-terraform/modules": Rel: can't make /Users/eddiebachle/git/stockx/data-ai-platform-terraform/modules relative to ../modules
 
The configured "remote" backend encountered an unexpected error. Sometimes this is caused by network connection problems, in which case you could retry the command. If the issue persists please open a support ticket to get help resolving the problem.

Our directory structure looks like:

total 32
drwxr-xr-x   8 eddiebachle  staff   256 Feb 23 19:30 ./
drwxr-xr-x  14 eddiebachle  staff   448 Feb 23 18:00 ../
drwxr-xr-x   6 eddiebachle  staff   192 Feb 23 19:30 .terraform/
-rw-r--r--   1 eddiebachle  staff  2503 Feb 23 19:03 .terraform.lock.hcl
-rw-r--r--   1 eddiebachle  staff   111 Feb 23 18:56 context.tf
lrwxr-xr-x   1 eddiebachle  staff    10 Feb 23 19:16 modules@ -> ../modules
-rw-r--r--   1 eddiebachle  staff   342 Feb 23 18:19 providers.tf
-rw-r--r--   1 eddiebachle  staff  1768 Feb 23 19:03 s3.tf

Trace log:

2023-02-23T19:35:11.547-0500 [INFO]  Terraform version: 1.3.9
2023-02-23T19:35:11.548-0500 [DEBUG] using github.com/hashicorp/go-tfe v1.9.0
2023-02-23T19:35:11.548-0500 [DEBUG] using github.com/hashicorp/hcl/v2 v2.16.0
2023-02-23T19:35:11.548-0500 [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2023-02-23T19:35:11.548-0500 [DEBUG] using github.com/hashicorp/terraform-svchost v0.0.0-20200729002733-f050f53b9734
2023-02-23T19:35:11.548-0500 [DEBUG] using github.com/zclconf/go-cty v1.12.1
2023-02-23T19:35:11.548-0500 [INFO]  Go runtime version: go1.19.6
2023-02-23T19:35:11.548-0500 [INFO]  CLI args: []string{"/opt/homebrew/Cellar/tfenv/3.0.0/versions/1.3.9/terraform", "plan"}
2023-02-23T19:35:11.548-0500 [TRACE] Stdout is a terminal of width 193
2023-02-23T19:35:11.548-0500 [TRACE] Stderr is a terminal of width 193
2023-02-23T19:35:11.548-0500 [TRACE] Stdin is a terminal
2023-02-23T19:35:11.548-0500 [DEBUG] Attempting to open CLI config file: /Users/eddiebachle/.terraformrc
2023-02-23T19:35:11.548-0500 [INFO]  Loading CLI configuration from /Users/eddiebachle/.terraformrc
2023-02-23T19:35:11.550-0500 [DEBUG] checking for credentials in "/Users/eddiebachle/.terraform.d/plugins"
2023-02-23T19:35:11.550-0500 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2023-02-23T19:35:11.550-0500 [DEBUG] will search for provider plugins in /Users/eddiebachle/.terraform.d/plugins
2023-02-23T19:35:11.550-0500 [DEBUG] ignoring non-existing provider search directory /Users/eddiebachle/Library/Application Support/io.terraform/plugins
2023-02-23T19:35:11.550-0500 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2023-02-23T19:35:11.553-0500 [INFO]  CLI command args: []string{"plan"}
2023-02-23T19:35:11.571-0500 [TRACE] Meta.Backend: built configuration for "remote" backend with hash value 4205472915
2023-02-23T19:35:11.572-0500 [TRACE] Preserving existing state lineage "4d81a3be-968e-b075-2f60-fe38aeed01cb"
2023-02-23T19:35:11.572-0500 [TRACE] Preserving existing state lineage "4d81a3be-968e-b075-2f60-fe38aeed01cb"
2023-02-23T19:35:11.573-0500 [TRACE] Meta.Backend: working directory was previously initialized for "remote" backend
2023-02-23T19:35:11.573-0500 [TRACE] Meta.Backend: using already-initialized, unchanged "remote" backend configuration
2023-02-23T19:35:11.575-0500 [DEBUG] Service discovery for app.terraform.io at https://app.terraform.io/.well-known/terraform.json
2023-02-23T19:35:11.576-0500 [TRACE] HTTP client GET request to https://app.terraform.io/.well-known/terraform.json
2023-02-23T19:35:11.815-0500 [DEBUG] Retrieve version constraints for service tfe.v2.1 and product terraform
2023-02-23T19:35:11.815-0500 [TRACE] HTTP client GET request to https://checkpoint-api.hashicorp.com/v1/versions/tfe.v2.1?product=terraform
2023-02-23T19:35:12.250-0500 [TRACE] Meta.Backend: instantiated backend of type *remote.Remote
2023-02-23T19:35:12.253-0500 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2023-02-23T19:35:12.255-0500 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/aws v4.55.0 for darwin_amd64 at .terraform/providers/registry.terraform.io/hashicorp/aws/4.55.0/darwin_amd64
2023-02-23T19:35:12.255-0500 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/time v0.9.1 for darwin_amd64 at .terraform/providers/registry.terraform.io/hashicorp/time/0.9.1/darwin_amd64
2023-02-23T19:35:12.255-0500 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/aws/4.55.0/darwin_amd64 as a candidate package for registry.terraform.io/hashicorp/aws 4.55.0
2023-02-23T19:35:12.255-0500 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/time/0.9.1/darwin_amd64 as a candidate package for registry.terraform.io/hashicorp/time 0.9.1
2023-02-23T19:35:13.632-0500 [TRACE] providercache.fillMetaCache: using cached result from previous scan of .terraform/providers
2023-02-23T19:35:13.680-0500 [DEBUG] checking for provisioner in "."
2023-02-23T19:35:13.680-0500 [DEBUG] checking for provisioner in "/opt/homebrew/Cellar/tfenv/3.0.0/versions/1.3.9"
2023-02-23T19:35:13.680-0500 [DEBUG] checking for provisioner in "/Users/eddiebachle/.terraform.d/plugins"
2023-02-23T19:35:13.680-0500 [TRACE] Meta.Backend: backend *remote.Remote supports operations
2023-02-23T19:35:13.808-0500 [INFO]  backend/remote: starting Plan operation

ebachle avatar Feb 24 '23 00:02 ebachle

Hey @apparentlymart, do you know if this is something that has eyes on it yet?

My org is struggling a bit here as we're pinned out on 1.2.9 right now because we use that symlinking strategy for getting modules into the working directory that the terraform CLI is running in so that they're in the tarball that goes to Terraform Cloud for remote runner.

We could come up with workarounds, but wanted to see if there was any traction on this issue first.

Thanks again for looking into this!

ebachle avatar Oct 27 '23 15:10 ebachle

Hi @ebachle, no update as far as I am aware although I will re-raise this in triage to see if your issue is the same as the original issue, and if it can be prioritized. I suspect that given only two people have raised this to date, it is not a high priority relative to other issues. Thanks!

crw avatar Oct 27 '23 22:10 crw