Backend S3: specify region in AWS CLI configuration instead of Terraform configuration

[SebTardif]

in terraform backed "s3" region should not be required since it can be provided in ~/.aws/credentials and ~/.aws/config

It's worst than it sound, because backed "s3" doesn't allow variable and so now I get my developers modifying that code, and end-up in git repository, just not workable.

terraform {
 backend "s3" {
 bucket = "sil-rancher-terraform-up-and-running-state"
 key = "terraform/PlatformOne-V1-rancher/terraform.tfstate"
 region = "us-gov-east-1"
 dynamodb_table = "terraform-up-and-running-locks"
 encrypt = true
 }
}

[apparentlymart]

Hi @SebTardif,

I believe the design assumption here is that the state storage region of a particular configuration won't change between runs, except in the rare case where you're explicitly migrating to a new region for some reason. If you had one person applying the configuration on a system with a different region than another then you could end up with a "forked" Terraform state, where there are two different latest state snapshots.

Given that, I think this is an intentional design decision and so I'm going to relabel this as an enhancement so we can discuss your underlying use-case further. To continue, it would be helpful if you could say more about this requirement for some of your developers to use a different region than what's represented in the configuration Then we can think about possible ways to meet that use-case either with features already in Terraform or with potential new features.

Thanks!

[SebTardif]

@apparentlymart We have a lot of of Terraform root modules, so lot of projects, lot of use of Terraform. What we ask or at least hope from our developers is only having to change terraform.tfvars when they change of environment, aws account, region.

The way our brain works, it's unexpected to have to specify the region more than one place, we already have it in the aws credential/config. In practice, we see that developer always un-expect to have to change again the region to a different file than terraform.tfvars when also it's already specified in their aws session.

You already know that specify the region is redundant. Why access_key is optional but not region?

aws configure ask 3 things, see https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

• AWS Access Key ID
• AWS Secret Access Key
• Default region name

Look at your AWS settings, did you set region?

[DJAlPee]

Just ran into this issue, too...

We are applying the same terraform config via our CI system to multiple AWS accounts. Until now, all are using the same "default" region, but there are now some AWS accounts, which are using using a different region.

Until now, the region was "hardcoded" in our CI by using AWS_DEFAULT_REGION environment property. I didn't know, that region is a mandatory attribute until I removed this hardcoded setting. At least this attribute is not treated as mandatory, because you can still omit it, as long as you provide the the region as environment property.

In our case the (default) region is determined during the "getting credentials" step in our CI pipeline, which writes the credentials (including the region) into the shared credentials files. Before the removal of the hardcoded environment property, we could just have our tf config like this:

terraform {
  backend "s3" {
    encrypt = true
  }

  [...]
}

and control the usage of the credentials within the CI "execution" step:

terraform init -backend-config="bucket=${BUCKET}" \
               -backend-config="key=terraform.tfstate" \
               -backend-config="profile=prod" \
               -backend=true \
               -get=true \
               -input=false

terraform apply -input=false -auto-approve -var-file="config.tfvars.json"

For my case, I have to code a workaround, which gets the region from the shared credential file and saves this as environment property.

In my opinion, region should be optional and treated the same way as the credentials! There is at least a bug in validating the terraform configuration file, because it is possible to omit a mandatory attribute.

Regarding the documentation, there is this "pattern" for partial configuration, but having this single attribute mandatory is conflicting this pattern and could be bypassed with region = "", which feels not right...

Fixing this "bug" should be done carefully, because I'm pretty sure, that this might break some configurations out there

[apparentlymart]

I was not involved in the original design of this backend and so I'm just speculating here but I believe the original design idea is to treat differently settings that relate to the storage location of the state (which must be consistent across all runs) vs. settings that in some sense describe who is running Terraform (which will typically be different for each user or execution environment).

The above is a general idea, but for the S3 backend in particular that division is like this:

• Storage location of the state: region, bucket, key
• Who is running Terraform: the access credentials, which typically come from either the environment or from the credentials file

I can see that this distinction feels a bit "murky" for region because in S3 bucket names are global namespace, and so there is never a situation where two different regions can have the same bucket name. Therefore in principle the S3 backend could treat region in particular as optional, infer it from the usual ambient SDK configuration sources if not specified, and return an explicit error if the specified bucket name belongs to the wrong region.

Would that address the use-case?

[DJAlPee]

Thank you for the clarification! Your suggestion should work for my case.

Maybe the way, how s3 backend is implemented, could be "inspired" by the implementation of s3_bucket_object resource.

There is also a way to "detect" the region of a bucket: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#GetBucketRegion

[apparentlymart]

Thanks for confirming, @DJAlPee!

The S3 backend is maintained by the AWS provider team rather than the Terraform CLI/Core team (despite also being in this repository) so ultimately that team will make the final call about what's best to do here but hopefully the information we gathered in this discussion will help them to make that decision when they are ready.

[gdavison]

In the case of the S3 backend, the region is part of the identification of the S3 bucket and the DynamoDB table, if used. It is also used for authentication, since AWS API connections require a region.

[roshan3133]

workaround which i used is to export AWS_DEFAULT_REGION="your region" before running terraform command and after this no need to specify region in terraform backend code

[gdavison]

@SebTardif, you can use either the environment variables AWS_REGION(preferred) or AWS_DEFAULT_REGION, as @roshan3133 mentioned.

The complication, however, is that region is used for two things here:

1. Authenticating with AWS
2. Specifying the location of the S3 bucket, and possibly DynamoDB table

AWS services are explicitly regional, though S3 also has a global namespace. This means that:

• The region of the S3 bucket can be determined using only the bucket name, since bucket names must be globally unique.
• The DynamoDB table, however, must consistently reference the table in a specific region. If the region specified for the backend is inadvertently changed, it will fail, either by not finding the table or by not having the correct checksum in the table.

We'll need to think about the best way to address region being used for two purposes. I'm going to leave this issue open to track work on it.

[a9db0]

This continues to be an issue in version 1.8.1. Despite the env variables and config file pointing to the intended target region, tf stubbornly looks in us-east-1 for the bucket. The error message even indicates which region the target bucket is in, but won't switch to pulling the state data from it. It is very frustrating. tf init desperately needs a simple command line switch to indicate the region for its state data bucket.