Find a way to bootstrap Consul ACL support

[remilapeyre]

https://github.com/terraform-providers/terraform-provider-consul/pull/60 introduces support for ACL with consul_acl_policy and consul_acl_token. Both require the ACL to have been previously bootstrapped manually to work.

Bootstrapping ACLs requires to keep track of the master token given by https://www.consul.io/api/acl/acl.html#bootstrap-acls so it would be written as clear text in the terraform state. The change in ACLs in Consul 1.4 was made so that the secret IDs would not need to be saved in unsecure situation and accessor IDs could be used instead.

Once ACLs are bootstrapped, the operator also needs to update the provider configuration and to set token or set the CONSUL_HTTP_TOKEN environment variable.

It would be nice to have a secure way to automate this.

Related: https://github.com/hashicorp/terraform/issues/9556

[frederikbosch]

Is there someone aware of a valid workaround? data "external"{} is re-run every time. Bootstrapping only needs to be executed once.

[remilapeyre]

Hi @frederikbosch, we don't have a satisfactory way to securely bootstrap the ACL support without the master token ending up in the Terraform state in plaintext.

Storing secrets in the Terraform state is a complicated issue, you can read more about it here : https://github.com/hashicorp/terraform/issues/516

Bootstrapping only needs to be executed once

Did you try using a provisioner (https://www.terraform.io/docs/provisioners/local-exec.html) ?

[frederikbosch]

@remilapeyre Exactly, I understand. I used local-exec indeed, combined with remote-exec to create policies, tokens and apply directly on the remote machines. I did that for Consul, Vault and Nomad. This did have the downside that the tokens ended-up in my (encrypted) remote S3 state. This is not a preferred solution, as indicated in the linked thread. However, manual bootstrapping is also not a preferred activity as alternative. I hope a better solution will become available soon.

[remilapeyre]

Hi everyone, I finally found a way to bootstrap the ACL system without having to write the management token in the Terraform state. https://github.com/hashicorp/vault/pull/10751 makes it possible to use Vault to do the ACL bootstrapping securely. Once it has been merged in Vault I will write a guide to show how to do this and generate additional Consul token from Vault using Terraform.

[lethargosapatheia]

Hello, @remilapeyre. Have you eventually written this guide? Is there anywhere we can see it?

[JalfResi]

I can see that Vault 1.11.0 has been released which contains:

• secrets/consul: Vault is now able to automatically bootstrap the Consul ACL system. [https://github.com/hashicorp/vault/pull/10751]

What changes are needed to the Consul provider to take advantage of this feature?

[techdragon]

Can this be done without Vault?

I don't really like the idea of being required to add (and manage) Vault if I want to be able to fully automate the setup and management of a Consul cluster.

[remilapeyre]

Hi, support for bootstrapping ACLs with Vault as indeed officially landed. While I haven't had time to write in the Terraform provider there is documentation in the Vault project: https://www.vaultproject.io/docs/secrets/consul#setup. You will be able to do the same thing using the Vault Terraform provider.

Can this be done without Vault?

I don't really like the idea of being required to add (and manage) Vault if I want to be able to fully automate the setup and management of a Consul cluster.

I understand that having an extra system to manage may not be ideal but it is very important for the solution to bootstrap the cluster to be secure, as all the rest of the applications security depends on this. I will continue to look for an alternative solution but cannot promise finding onem nor a timeline.

If you manage a small number of clusters I suppose doing manually might be best, or creating a small Vault cluster if you have a large number of Consul systems (one Vault cluster can bootstrap the ACLs of all your Consul clusters).

If you are okay with an insecure solution, you should be able to use en external resource to do this in the meantime.

[vortarian]

We did it this way, looking forward to it being supported in the official provider (vault 1.11.0 is min version):

resource "vault_mount" "consul" {
  path = "consul"
  type = "consul"
}

resource "vault_generic_secret" "consul_config" {
  depends_on = [vault_mount.consul]
  data_json = jsonencode({
    address = "http://consul.service.consul:8500",
  })
  path      = "${vault_mount.consul.path}/config/access"
}