Support `prefix` configuration in s3 backend

[vincer]

trafficstars

Terraform Version

1.7.0

Use Cases

Allow for simplified configuration of S3 backend when a prefix per project is wanted.

Attempted Solutions

Currently the only way to have the same prefix for both the default state and all non-default environments is to specify it twice:

bucket = "my-bucket"
key = "foo/terraform.tfstate"
workspace_key_prefix = "foo/env:"

This is an easy opportunity for mistakes to happen, particularly since variables/locals are not allowed in the backend configuration.

Proposal

Allow for a prefix configuration:

bucket = "my-bucket"
prefix = "foo/"
key = "terraform.tfstate"

This would be equivalent to the currently working example and result in the default state to reside at foo/terraform.tfstate and any non-default state to reside at foo/env:/<workspace>/terraform.tfstate.

References

No response

[apparentlymart]

Thanks for suggesting this, @vincer.

In principle we could use the addition of this new option as an opportunity to simplify the design further, on an opt-in basis, by removing the special treatment of the "default" workspace that is really there only because this backend was around before Terraform had support for a backend offering multiple workspaces.

Specifically, I mean that if and only if prefix is set:

• key and workspace_key_prefix are both forbidden. That is, an author must choose to either use prefix or the key/workspace_key_prefix pair, not both at the same time.
• When prefix is provided, it must end with a slash.
• When prefix is provided, the full object key for the state of a particular workspace is prefix + workspaceName + ".tfstate", regardless of the workspace name.
• When prefix is provided, the operation of listing all of the existing workspaces is implemented as finding all of the keys with the specified prefix, ignoring any whose full path does not end with .tfstate.

This would make the layout in the S3 bucket far more uniform than the current approach: default is no longer a special case, there's no need for the weird env: portion to distinguish non-default workspaces, and because this follows a more conventional layout for S3 bucket objects the set of workspaces for a particular configuration will appear as a "folder" in the S3 console:

• prefix/default.tfstate
• prefix/other.tfstate

Ultimately the design of the S3 backend is up to the AWS provider team, which I'm not a member of, so I mean the above only as a slight variation on the design as proposed which the AWS provider team might consider when deciding what to do with this proposal. However, I'm curious if you (@vincer) think that this variation would work to meet your goals, or if I've missed supporting something important that your proposal would have supported.

[vincer]

Hi @apparentlymart !

Yes, that sounds even better to me. In my proposal, I was trying to keep things as backwards compatible as possible while fixing my main nit (the lack of DRYness in the config), but simplifying the S3-side structure is a nice improvement as well IMO.

Logistically, does this request stay here or does it somehow get moved to AWS provider's issues?

[apparentlymart]

Thanks @vincer!

If we did choose to do something like what I proposed, I think a part of it would be to make sure that there's some reasonable way to migrate from the old way to the new way without losing data. I'm not sure if that's actually possible with today's backend design -- it tends to assume that the old and new backends are separate when migrating, rather than having only a subset of the workspaces move physical storage -- and so it might well be that it turns out to be infeasible to do what I proposed, but we'll see.

Although the AWS provider team are the people who would probably work on this, the change would still need to be in this codebase and so this is the best place for this issue to sit. The backend/s3 label is there to make it more visible to the folks on that team.