Unable to pass X.509 credentials to Vault via Terraform

[bdhayco]

Hi there,

Terraform - 1.0.8 & 1.0.7 Vault - 1.8.3-1 (linux) terraform-provider-vault - 2.23.0 & 2.22.0 terraform-provider-tls - 3.1.0 terraform-provider-keycloak - 3.4.0

I have been having an issue with terraform-provider-vault where the local versions of the PKI bundle are not being passed back to the CA (root or Int).

For the root CA I am using the hashicorp provider TLS to generate the root key and the ca-cert The keypair for both the root & Intermediate CA's can be writen to local files and manually import them into vault, either by the CLI or the GUI. I then use "vault_pki_secret_backend_config_ca" and "vault_pki_secret_backend_intermediate_set_signed" to attempt to write the CA credentials back to the root CA without success.

  resource "vault_mount" "root" {
    type                      = "pki"
    path                      = "pki-root-ca"
    default_lease_ttl_seconds = 31556952  # 1 years
    max_lease_ttl_seconds     = 157680000 # 5 years
    description               = "Root Certificate Authority"
  }
  resource "tls_private_key" "ca_key" {
    algorithm = "RSA"
    rsa_bits = 4096
  } 
  
  resource "tls_self_signed_cert" "ca_cert" {
    private_key_pem = tls_private_key.ca_key.private_key_pem
    key_algorithm = "RSA"
    subject {
      common_name = "Root CA"
      organization = "BOB"
      organizational_unit = "BOB" 
      country = "AU"
    }
    # 175200 = 20 years
    validity_period_hours = 175200
    allowed_uses = [
      "cert_signing",
      "crl_signing"
    ]
    is_ca_certificate = true 
  }

  resource "local_file" "root_ca_pem_bundle" {
     sensitive_content = "${tls_private_key.ca_key.private_key_pem}${tls_self_signed_cert.ca_cert.cert_pem}"
     filename = "${path.root}/output/root_ca/ca_pem_bundle.pem"
     file_permission = "0777"
   }
  
  resource "vault_pki_secret_backend_intermediate_set_signed" "root_ca" {
    depends_on = [
       tls_private_key.ca_key,
       tls_self_signed_cert.ca_cert
    ] 
    backend = vault_mount.root.path
    certificate = local_file.root_ca_pem_bundle.sensitive_content
  }

  resource "vault_pki_secret_backend_config_ca" "root_ca_config" {
    depends_on = [ vault_mount.root, tls_private_key.ca_key]  
    backend  = vault_mount.root.path
    pem_bundle = local_file.root_ca_pem_bundle.sensitive_content
  }

The Intermediate CA server

  resource "vault_mount" "pki_int" {
    path                      = "pki-intermediate"
    type                      = "pki"
    description               = "This is the intermediate CA"
    default_lease_ttl_seconds = 3600
    max_lease_ttl_seconds     = 86400
  }

  resource "vault_pki_secret_backend_intermediate_cert_request" "intermediate" {
    depends_on = [vault_mount.pki_int]
    backend    = vault_mount.pki_int.path
    type       = "exported" #'exported/internal'
  # This appears to be overwritten when the CA signs this cert, I'm not sure 
  # the importance of common_name here.
    common_name        = "Intermediate Certificate"
    format             = "pem"
    private_key_format = "der"
    key_type           = "rsa"
    key_bits           = "4096"
}

  resource "vault_pki_secret_backend_root_sign_intermediate" "intermediate" {
     depends_on = [ 
        tls_self_signed_cert.ca_cert
      ]
    backend = vault_mount.root.path

    csr                  = vault_pki_secret_backend_intermediate_cert_request.intermediate.csr
    common_name          = "Intermediate Certificate"
    exclude_cn_from_sans = true
    ou                   = "BOB"
    organization         = "BOB"
    country              = "AU"
    # Note that I am asking for 8 years here, since the vault_mount.root has a max_lease_ttl of 5 years
    # this 8 year request is shortened to 5.
    ttl = 252288000 #8 years
  }

  resource "vault_pki_secret_backend_config_ca" "int_ca_config" {
    depends_on = [ 
      vault_mount.pki_int
    ]  
    backend  = vault_mount.pki_int.path
    pem_bundle = local_file.ca_int_pem_bundle.sensitive_content
  }

  resource "vault_pki_secret_backend_intermediate_set_signed" "intermediate" {
    depends_on = [
      vault_mount.pki_int
    ]
    backend     = vault_mount.pki_int.path
    certificate = "${vault_pki_secret_backend_root_sign_intermediate.intermediate.certificate}"
  }

When I run this code, in amongst the other CA code, I do not get any logging references relating back to it. I should be seeing the string "/${instance}/config/ca" in the log file, and I have Terraform running under DEBUG logging. The only reference I find in that to 'root_ca_config' is either Provider or Reference Transformer entries, and a single entry for pruneUnusedNodes: vault_pki_secret_backend_config_ca.root_ca_config (expand) is no longer needed, removing

Looking in the terraform output when running I can see I see no entries in relation to "vault_pki_secret_backend_config_ca" code running. Occasionally, I may see a reference to the the entry stating that the Intermediate has been refreshed. vault_pki_secret_backend_config_ca.int_ca_config: Refreshing state... [id=pki-intermediate] vault_pki_secret_backend_intermediate_set_signed.intermediate: Refreshing state... [id=pki-intermediate/intermediate/set-signed]

But nothing for the Root CA.

I can, after the code completes, manually add the bundle file to the root CA and it accepts the bundle. But when I rerun the code the same behavior occurs with the Intermediate CA, in that vault_pki_secret_backend_config_ca.int_ca_config, or vault_pki_secret_backend_intermediate_set_signed.intermediate Don't run, although the do get refreshed. Again there is nothing in the logs.

The signed Intermediate cert and key (if I exported it) not being added to the CA. Internal/exported on the private key doesn't make a difference. Debug Output none

Expected Behavior

For both the Root and Int CA's I would expect to see the x.509 bundle uploaded to the appropriate CA. Currently I am not seeing this behavior. I can manually create a CA using the Vault CLI, using the same credentials. There have been similar issues reported previously.

Actual Behavior The vault_pki_secret_backend_intermediate_set_signed or vault_pki_secret_backend_config_ca.int_ca_config code stanzas are not being run, as such the X.509 keypair are not being added to the CA server How ever randomly and in no pattern I have been able to determine, the 'vault_pki_secret_backend_intermediate_set_signed' appears to run, if this happens it is after I have manually applied the root bundle to the root CA. I reset vault (stop vault, delete all data in the vault data path, delete all logs, restart vault, reinitialize) and then run the code, with no changes. It will fails.

If after a terraform run, I run the commands vault write pki-root-ca/config/ca pem_bundle=@output/root_ca/ca_pem_bundle.pem Success! Data written to: pki-root-ca/config/ca vault write pki-intermediate/config/ca pem_bundle=@output/int_ca/int_ca_pem_bundle.pem Success! Data written to: pki-intermediate/config/ca vault write pki-intermediate/intermediate/set-signed certificate=@output/int_ca/int_cert.pem Success! Data written to: pki-intermediate/intermediate/set-signed

But the same code in terraform doesn't appear to run.

References: https://github.com/hashicorp/terraform-provider-vault/blob/e349937c55f143b1ff9a8d894116da3c55ff72d0/vault/provider.go https://github.com/hashicorp/terraform-provider-vault/pull/158 https://registry.terraform.io/providers/hashicorp/vault/latest/docs/resources/pki_secret_backend_intermediate_set_signed https://medium.com/@stvdilln/creating-a-certificate-authority-with-hashicorp-vault-and-terraform-4d9ddad31118

Important Information This is a module that will take a newly initialized instance of vault and configure it to create a root/Int CA's and to create a basic number of Vault & CA Admin roles in Keycloak in a terraformed managed realm.

[geekofalltrades]

I reset vault (stop vault, delete all data in the vault data path, delete all logs, restart vault, reinitialize) and then run the code, with no changes. It will fails.

I think this is expected behavior, but deeply annoying. The API endpoints backing this resource are write-only. They don't support read, and they don't support delete. You can see this reflected in the implementation of the resource: the read and delete functions are empty stubs. This means that, once these resources are in the Terraform state, Terraform has no way of detecting drift, and can't detect when they need to be updated.

If you terraform taint vault_pki_secret_backend_config_ca.root_ca_config and then apply, does it work once?

[geekofalltrades]

I was able to work around this with a combination of null_resource and replace_triggered_by.

resource "vault_mount" "pki" {
  path = "pki"
  type = "pki"
}

resource "null_resource" "pki_backend_tracker" {
  triggers = {
    accessor = vault_mount.pki.accessor
  }
}

resource "vault_pki_secret_backend_config_ca" "pki" {
  lifecycle {
    replace_triggered_by = [null_resource.pki_backend_tracker]
  }

  backend    = vault_mount.pki.path
  pem_bundle = local.pki
}

The null_resource is necessary as a shim because it plans to replace itself if the PKI backend mount changes, which triggers the replace_triggered_by on the config_ca resource. Otherwise, if the Vault backend state is wiped, then the PKI mount is planned as a create, which doesn't trigger the replace_triggered_by.

[anarsen]

@geekofalltrades Thank you for this. I ran into a similar issue, however mine being with the fact that the vault_pki_secret_backend_intermediate_cert_request resources weren't recreated when my PKI mounts were re-created.

I mistakenly thought the replace_triggered_by worked if I supplied the vault_mount.x.accessor but no. The null_resource & triggers combo did the trick though!