Cannot pull modules from S3 buckets in AWS China regions

[julian7]

Terraform Version

Terraform v1.0.5
on darwin_amd64

Terraform Configuration Files

module "example" {
    source = "s3::https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip"
}

Debug Output

...
Initializing modules...
2021-08-19T08:48:35.420+0200 [TRACE] ModuleInstaller: installing child modules for . into .terraform/modules
2021-08-19T08:48:35.421+0200 [DEBUG] Module installer: begin example
2021-08-19T08:48:35.421+0200 [TRACE] ModuleInstaller: example is not yet installed
2021-08-19T08:48:35.421+0200 [TRACE] ModuleInstaller: cleaning directory .terraform/modules/example prior to install of example
2021-08-19T08:48:35.421+0200 [TRACE] ModuleInstaller: example address "s3::https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip" will be handled by go-getter
Downloading s3::https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip for example...
2021-08-19T08:48:35.421+0200 [DEBUG] will download "s3::https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip" to .terraform/modules/example
2021-08-19T08:48:35.421+0200 [TRACE] fetching "s3::https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip" to ".terraform/modules/example"
2021-08-19T08:48:35.728+0200 [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json

(sry, considered making a gist, but for ten lines?)

Crash Output

N/A

Expected Behavior

I expected the example to fail because of invalid S3 bucket or nonexisting file. I expect terraform to download modules correctly when correct source is present.

Actual Behavior

terraform init returns the following error:

│ Error: Failed to download module
│
│ Could not download module "example" (main.tf:1) source code from "s3::https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip": RequestError: send request failed
│ caused by: Get "https://s3.amazonaws.amazonaws.com/s3/example-bucket/example.zip": dial tcp: lookup s3.amazonaws.amazonaws.com: no such host

The host name s3.amazonaws.amazonaws.com is totally not what the original URL was.

Steps to Reproduce

terraform init

Additional Context

I've updated go-getter at my end to honor amazonaws.com.cn S3 URLs (see references), and it works for me.

References

• hashicorp/go-getter#128
• hashicorp/go-getter#335

[ewbankkit]

@julian7 It looks like there is a problem with your module's source URL: s3::https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip. According to the list of supported protocols the URL should be either s3://example-bucket/example.zip or https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip.

[julian7]

@ewbankkit I couldn't see the s3://example-bucket/example.zip format anywhere in the list of supported protocols. If I'm not mistaken, prefixing the URL with s3:: is just forcing the protocol, and it can be any (read: any if it's HTTP) protocol underneath.

I've tested both of your suggestions on AWS-CN in cn-northwest-1 region:

Downloading s3://example-bucket/example.zip for example...
╷
│ Error: Failed to download module
│
│ Could not download module "example" (test.tf:1) source code from "s3://example-bucket/example.zip": URL is not a valid S3 complaint URL

Downloading https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip for example...
╷
│ Error: Failed to download module
│
│ Could not download module "example" (test.tf:1) source code from "https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip": Get
│ "https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip": 301 response missing Location header

While, if I use the original test with my patched version, I'm getting the following error:

Downloading s3::https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip for example...
╷
│ Error: Failed to download module
│
│ Could not download module "example" (test.tf:1) source code from "s3::https://s3.cn-north-1.amazonaws.com.cn/example-bucket/example.zip": BucketRegionError: incorrect region, the
│ bucket is not in 'cn-north-1' region at endpoint ''
│ 	status code: 301, request id: , host id:

This latter has merit, or at least nobody has ever created example-bucket in cn-north-1 yet.

[apparentlymart]

It does seem reasonable that go-getter would be the cause of this, because it has its special "detector" behavior that tries to recognize and normalize the given URL, but the current version of that seems too eager in its normalization and only supports the URL schemes supported by the main AWS partition, and not the other separated partitions like China and GovCloud.

A logistical problem here is that go-getter doesn't have any dedicated maintainers, and yet go-getter is a shared dependency across many HashiCorp products and so we'll need to be careful about merging a change to it. Something like what's proposed in hashicorp/go-getter#335 does seem plausible to me, though I wonder if someone with more knowledge about the alternative AWS partitions can comment on whether there's a more generalized way to do this that would also work for GovCloud, or perhaps correct me that GovCloud already works due to it having a similar enough URL scheme to the primary partition. (Ideally I'd prefer a solution that won't need ongoing additions as AWS introduces new partitions or URL schemes, but I don't know if that's actually a feasible requirement to meet.) :thinking:

[julian7]

@apparentlymart for the record, here are the S3 endpoint URLs: https://docs.aws.amazon.com/general/latest/gr/s3.html#s3_region

[julian7]

Also, I was able to use the following source setting (which I don't know whether it's mentioned anywhere) with vanilla terraform v1.0.5 CLI:

module "example" {
  source = "s3::/example-bucket/path/to/example.zip?region=cn-north-1"
  ...
}

[apparentlymart]

Thanks for sharing the docs and that other example, @julian7!

I'm honestly a bit baffled as to how that last example is working, because I'd expect it to be asking the S3 getter to fetch file:///example-bucket/path/to/example.zip?region=cn-north-1, which doesn't really make any sense. I guess it must be falling into the fallback codepath, and just kinda magically working out because that codepath ignores the scheme of the URL and only looks at the path portion, and so it doesn't really matter that it's a file:// URL.

Go-getter is generally far too clever for its own good, but I'm glad you found something that works, at least. There isn't any test case shaped like that in go-getter so I have to assume this is by accident rather than by design, but since it's covered by our compatibility promises we can't change it now and ought to go add some additional test coverage to go-getter for it.

[apparentlymart]

(Incidentally, the forthcoming Terraform v1.1 includes an improvement to make terraform init report back the result of go-getter's detection in its status messages, instead of just echoing back the string you wrote, and so I think it would've at least then admitted in the UI that it was using the wrong URL in the original case here, rather than the confusing situation you saw where the status message said one URL but the error message reported a completely different one.)

[julian7]

@apparentlymart you're right, it makes no sense. However, I think at this stage the best what we can do is to use whatever is working, it's very hard to add more features to an already convoluted system. Especially, when there is a solid workaround. I'd say the workaround should be documented instead of any code changes at the moment.