Would you be open to terraform state backend in Vault?

[gitirabassi]

I've read your Contributing.md and before opening a PR I would like to ask if you would consider a terraform backend with locking in Vault KV (both v1 and v2). I know you're working hard on some refactoring for the backend state, but would this feature be accepted if completed and submitted as a PR?

[crw]

Hi @gitirabassi, thanks for the question. I will add this to the issue triage queue to see if we can get an answer for you.

[crw]

Hi @gitirabassi, we are not currently considering new Terraform backends. I'll update this issue if that changes.

In the meantime, could you please share your use case for storing Terraform state in Vault? On the surface, this does not seem like a natural solution simply due to the size of a Terraform state file potentially being quite large, and Vault's main use case is to store relatively smaller-sized secrets. Thanks for any feedback on this!

[gitirabassi]

I figured as much given the note on the CONTRIBUTING.md but worth asking. The best tool to configure Vault at this time is Terraform but configuring Vault without leaking any secret in the state is basically impossible in any real world scenario. So while configuring Vault would be good to store the state in a place that can be very easily secured from prying eyes. the Terraform user could have its own KV secret engine and the only one allowed access to it. Does it make sense? do you have a solution to this problem already?

[gitirabassi]

Do you mind keeping this issue open (for others to refer to while the backend system gets reworked)

[crw]

Hi @gitirabassi, I did a little more digging and found this issue: https://github.com/hashicorp/terraform/issues/16066 -- it is very old, but I think it would be helpful to add your use case to that issue and close this one as a duplicate. What do you think?

There is a Consul backend, however it is unlikely to meet your requirements. https://www.terraform.io/language/settings/backends/consul

[tjad]

I know this does not really apply to Terraform per se, however my thoughts regarding #16066 is that the idea of vault not being built for large resources a bit silly. How large can a state file get ? And why should Vault be limited in size. If I'm going to use Vault for storing secrets/sensitive data, it should cover all use cases. State file can't possibly be GB or even 10s of MB ?

To me it the response doesn't seem valid. We should be allowed to use Vault as a backend. If Vault is not an option, then Hashicorp did wrong. This may seem like strong criticism, but think about this objectively. I don't want to maintain permissions to sensitive data on several systems, I want to use Vault as a one stop for sensitive data and configure its permissions accordingly. It's too easy to miss permission configuration when coordinating/synchronising secrets - especially if they are the same secrets stored in the vault, and just so happen to appear in the state file.

[drewhammond]

Every year or so I poke around looking for updates on this, and I've always thought this was a huge missed opportunity for Hashicorp because enterprise customers would pay for this.

We have sensitive data that we don't want to ever exist in plaintext in a state file, and Terraform hasn't been able to solve that problem in a satisfactory way. It means that we have to build clunky tooling around Terraform or choose to handle this outside of Terraform which hurts adoption.

Perhaps naively, I'd think one of these two things should definitely be possible:

• Terraform should be able to encrypt/decrypt sensitive data in the state file using a key that is read from Vault at runtime (https://github.com/hashicorp/terraform/issues/9556)
• Terraform should be able to store the entire state file in Vault (as requested in this issue)

Clearly it isn't trivial since they seem so simple and yet we still haven't seen progress on either approach.

I'll reach out to our account rep, but I hope Hashicorp will consider this for OSS too.

[atrull]

surely the json blob of a kvv2 would be fine ? has anyone tried it ?

[crw]

I am going to close this one as a dupe of https://github.com/hashicorp/terraform/issues/16066. Please feel free to continue on-topic conversation and use cases in that issue. Thanks!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.