PKI - Add missing parameters in some resources

[sestegra]

In general, the following scenarios are not supported:

• the managed keys by a KMS
• the new issuer API (from Vault 1.11)

Regarding official documentation, following parameters or possible values are missing:

For generate root and intermediate

• type: the possible values existing and kms are missing
• issuer_name
• key_name
• key_ref
• managed_key_name
• managed_key_id
• not_before_duration
• not_after

For sign and role

• issuer_ref

[benashz]

Hi @sestegra ,

The managed keys feature will be coming out in the 3.9.0 release. As for the new Issuer PKI support, that should come out soon after 3.9.0.

Thanks,

Ben

[krarey]

In addition to adding support for issuer_ref on the noted resources, we should also add a resource for calling the root/replace and config/issuers endpoints to update the associated root CA certificate and default issuer associations: https://developer.hashicorp.com/vault/api-docs/secret/pki?optInFrom=vault-io#set-issuers-configuration

[krarey]

Similarly, vault_pki_secret_backend_config_ca should return the imported_issuers attribute from the Vault response, to chain into the aforementioned issuer configuration.

[lboynton]

Hi @benashz, any plans to release the new Issuer PKI support soon? It would be great to be able to specify the default issuer through terraform.

[itspngu]

This is a huge pain point, I'm not quite sure why there's not more people voicing their interest in this feature, as without it using Terraform to configure a Vault >= 1.11 PKI mount is pretty much impossible (once you start rotating CAs).

I messed with using HTTP data sources and vault_generic_endpoint as a workaround but came to realize the /issuers API endpoint returns issuer IDs in alphabetical order, not chronologically. I ended up having to write a rather elaborate provisioner script as a stop-gap solution, leaving it here in hopes of sparing some people the headaches I've had with this.

Click to expand

#!/usr/bin/env python3

# Given the path of a Vault PKI secrets engine mount, this script configures the mount's
# "default issuer" to match a specified certificate serial number. This is currently not
# possible using Terraform, but needed for PKI engines on Vault >= 1.11 in order for rotation
# to work. Use as local-exec provisioner on a pki_secret_backend_intermediate_set_signed resource.
#
# ref: https://developer.hashicorp.com/vault/docs/secrets/pki/considerations#always-configure-a-default-issuer
# ref: https://github.com/hashicorp/terraform-provider-vault/issues/1510
# ref: https://developer.hashicorp.com/vault/api-docs/secret/pki#managing-keys-and-issuers
#
# There's no error handling whatsoever in this script because hopefully it can be removed soon.


import json
import os
import requests
import sys

from cryptography import x509
from cryptography.x509.oid import NameOID


if __name__ == "__main__":

    query = json.loads(sys.stdin.read())

    vault_addr = os.getenv("VAULT_ADDR")
    if not vault_addr:
        print("`VAULT_ADDR` not set, aborting.", file=sys.stderr)
        sys.exit(1)

    vault_token = os.getenv("VAULT_TOKEN")
    if not vault_token:
        print("`VAULT_TOKEN` not set, aborting.", file=sys.stderr)
        sys.exit(1)

    vault_cacert = os.getenv("VAULT_CACERT")
    if not vault_cacert:
        print("`VAULT_CACERT` not set, aborting.", file=sys.stderr)
        sys.exit(1)

    mount_path = query.get("mount_path")
    if not mount_path:
        print("Missing `mount_path` in query data, aborting.", file=sys.stderr)
        sys.exit(1)

    desired_issuer_id = None # This is what we're trying to figure out
    desired_serial_number = query.get("serial_number")
    if not desired_serial_number:
        print("Missing `serial_number` in query data, aborting.", file=sys.stderr)
        sys.exit(1)

    base_url = f"{vault_addr}/v1/{mount_path}"
    headers = {
        "X-Vault-Token": vault_token,
        "X-Vault-Request": "true",
    }

    # Get a list of all issuers in this mount
    # ref: https://developer.hashicorp.com/vault/api-docs/secret/pki#list-issuers
    with requests.get(f"{base_url}/issuers?list=true", headers=headers, verify=vault_cacert) as r:
        res = r.json()
        issuer_ids = res.get("data").get("keys")

    # Iterate over issuers and read each issuer's certificate
    for issuer_id in issuer_ids:
        # ref: https://developer.hashicorp.com/vault/api-docs/secret/pki#read-issuer-certificate
        with requests.get(f"{base_url}/issuer/{issuer_id}", headers=headers, verify=vault_cacert) as r:
            res = r.json()
            issuer_cert_pem = res.get("data").get("certificate")

            # Load certificate
            issuer_cert = x509.load_pem_x509_certificate(issuer_cert_pem.encode())

            # Get its serial number and encode its hexadecimal representation in a string
            issuer_cert_serial_number = '%x' % issuer_cert.serial_number
            # Add colons because... colons (why is this x509 library returning it as an int to begin with?)
            issuer_cert_serial_number_hex = ":".join(issuer_cert_serial_number[i:i+2] for i in range(0, len(issuer_cert_serial_number), 2))

            # Compare serial number against desired `serial_number` passed from Terraform
            if issuer_cert_serial_number_hex == desired_serial_number:
                # If they match, take note and quit the loop.
                desired_issuer_id = issuer_id
                break

    # Have we found what we're looking for?
    if not desired_issuer_id:
        print(f"Unable to find issuer for certificate with serial number {desired_serial_number}, aborting.", file=sys.stderr)
        sys.exit(1)

    # Set new default issuer and live happily ever after. Maybe.
    # ref: https://developer.hashicorp.com/vault/api-docs/secret/pki#set-issuers-configuration
    issuer_config = {
        "default": issuer_id,
        # Whatever this is supposed to mean, it doesn't mean what the name implies
        "default_follows_latest_issuer": True
    }
    with requests.post(f"{base_url}/config/issuers", data=issuer_config, headers=headers, verify=vault_cacert) as r:
        res = r.json()

        # The response JSON should contain the issuer ID we just set.
        configured_issuer_id = res.get("data").get("default")
        if configured_issuer_id != desired_issuer_id:
            print(f"Updating issuer config failed, configured issuer {configured_issuer_id} does not match desired issuer {desired_issuer_id}.", file=sys.stderr)
            sys.exit(1)

    print(f"Updated issuer of PKI secrets engine mount '{mount_path}' to {configured_issuer_id}")
    sys.exit(0)

Use like this:

resource "vault_pki_secret_backend_intermediate_cert_request" "sub" {
  backend = "path/to/mount"
  type     = "internal" # Do not return private key, do not save it in tfstate, do not collect $400

  common_name  = "HashiCups"
}
resource "vault_pki_secret_backend_root_sign_intermediate" "sub" {
  backend = "path/to/issuer"

  csr = vault_pki_secret_backend_intermediate_cert_request.sub.csr
}
locals {
  provisioner_query = jsonencode({
    mount_path    = vault_pki_secret_backend_intermediate_cert_request.sub.backend
    serial_number = vault_pki_secret_backend_root_sign_intermediate.sub.serial_number
  })
}
resource "vault_pki_secret_backend_intermediate_set_signed" "sub" {
  backend     = vault_pki_secret_backend_intermediate_cert_request.sub.backend
  certificate = vault_pki_secret_backend_root_sign_intermediate.sub.certificate

  provisioner "local-exec" {
    interpreter = ["/bin/sh", "-c"]
    command     = "echo '${local.provisioner_query}' | /usr/bin/env python3 -u '${path.module}/scripts/set-default-issuer.py'"
  }
}

When rotating, the issuer is fixed, set, corrected, configured, whatever you'd like to call it:

module.pki_v1.vault_pki_secret_backend_intermediate_set_signed.sub: Creating...
module.pki_v1.vault_pki_secret_backend_intermediate_set_signed.sub: Provisioning with 'local-exec'...
module.pki_v1.vault_pki_secret_backend_intermediate_set_signed.sub (local-exec): Executing: ["/bin/sh" "-c" "echo '{\"mount_path\":\"path/to/mount\",\"serial_number\":\"7c:e4:42:06:a6:cd:bf:be:94:0b:eb:0c:37:64:9b:98:49:47:3d:d1\"}' | /usr/bin/env python3 -u '../../modules/vault-pki-sub-ca/scripts/set-default-issuer.py'"]
module.pki_v1.vault_pki_secret_backend_intermediate_set_signed.sub (local-exec): Updated issuer of PKI secrets engine mount 'path/to/mount' to f0b25918-b96a-2353-236c-c5e88a8d8071
module.pki_v1.vault_pki_secret_backend_intermediate_set_signed.sub: Creation complete after 1s [id=path/to/mount/intermediate/set-signed]

[geekofalltrades]

The new issuer feature in Vault 1.11 is 7 months old. This lack of first-class support in the Terraform provider is embarrassing.

Here's what I've done to keep the pre-1.11 behavior in the PKI backend. You need at least Vault 1.11.6 or 1.12.2 for the default_follows_latest_issuer option. If you don't have those versions, the option is silently ignored and you will have a bad time.

resource "vault_mount" "pki" {
  path = "pki"
  type = "pki"
}

# XXX; The vault_pki_secret_backend_config_ca resource has no drift detection,
# as the API it wraps is write-only. If this resource persists in the state
# while Vault state is wiped underneath it, Terraform will not correctly
# detect that it needs to be recreated.
#
# lifecycle.replace_triggered_by on the config_ca doesn't solve this by itself,
# as when Vault state is wiped, vault_mount.pki is planned to be created, not
# updated or replaced, and resource creation doesn't trigger
# replace_triggered_by.
#
# To get the behavior we want, use this null_resource as a shim. It survives
# Vault state being wiped, and plans to replace itself if the accessor ID of the
# PKI mount changes. This replacement can be used to trigger
# replace_triggered_by on the config_ca resource.
resource "null_resource" "pki_backend_tracker" {
  triggers = {
    accessor = vault_mount.pki.accessor
  }
}

resource "vault_pki_secret_backend_config_ca" "pki" {
  lifecycle {
    replace_triggered_by = [null_resource.pki_backend_tracker]
  }

  backend    = vault_mount.pki.path
  pem_bundle = local.pki
}

# Vault 1.11 added new functionality in the pki backend in which there can be
# multiple issuing CAs ("issuers") installed in one mount. Adding a new CA cert
# now creates a new issuer, but doesn't update the backend to issue from that
# new issuer by default. The "default_follows_latest_issuer" option preserves
# the pre-1.11 behavior of always issuing from the last installed issuer.
#
# https://developer.hashicorp.com/vault/api-docs/secret/pki#notice-about-new-multi-issuer-functionality
# https://developer.hashicorp.com/vault/api-docs/secret/pki#default_follows_latest_issuer
#
# The config/issuers endpoint requires that we provide an issuer to set as the
# default in its "issuer" field, so first, figure out what the current default
# is and just pass it back. We can't do this until an issuer is actually
# installed, so we depends_on the pki config CA being installed. This is fine as
# long as we're not changing the CA in the same run we're first applying this
# config, as that would create a new issuer which isn't the default and then
# that issuer would get orphaned. Make sure this resource is applied on the
# current CA before rotating to a new CA.
data "vault_generic_secret" "pki_default_issuer" {
  depends_on = [vault_pki_secret_backend_config_ca.pki]

  path = "${vault_mount.pki.path}/issuer/default"
}

resource "vault_generic_endpoint" "pki_config_issuers" {
  path = "${vault_mount.pki.path}/config/issuers"
  data_json = jsonencode({
    default                       = data.vault_generic_secret.pki_default_issuer.data.issuer_id,
    default_follows_latest_issuer = true,
  })
  disable_delete = true
}

[ingvarch]

Any updates here ? We really need this feature. Still waiting.

[japtain-cack]

I am in need of these changes as well.

[camarrone]

The Vault provider should be in line with the vault API.

As said above by @itspngu, this is a huge pain point and my case it is a show stopper and a breaking point.

We can't keep writing alternative solutions and be more error prone due to lack of support on this provider.

Any ETC on this?

Thanks.

[vinay-gopalan]

Hey folks, apologies for the delayed response! 🙏🏼

We 100% agree that the TFVP should be in line with Vault's API! While Vault's feature set development continues to scale with multiple teams, the TFVP team is also trying to improve upon the long term sustainability of this project and scale this project's development by collaborating with more teams and contributors. This collaborative effort has growing pains and is slow, so again we apologize and appreciate the patience. 🙇🏼

In that context, the TFVP team is collaborating w/ the Vault CryptoSec team to appropriately support the multi-issuer feature set within the TFVP 😄 At present the development for this feature is on track w/ Vault's upcoming 1.14 release in mid-June; we'll try our best for an earlier release! I'll be sure to continue updating this thread w/ any relevant info, and thanks once again for the patience!

[nickschoenbaechler]

Is mid-June still the target? I'm trying to decide if it's best for me to work around this or to expect that TFVP will be ready in the next week or so.

[vinay-gopalan]

@nickschoenbaechler, yes, we are still targeting mid-June for the release of this feature. We're currently testing a feature branch on our end.

[vinay-gopalan]

Hey folks, just wanted to drop an update here that we released Multi-Issuer Functionality for PKI in the TFVP as part of v3.17.0. We apologize for the delay on the feature parity and appreciate your patience!

Going to close this issue for now, but please let us know of if you face any issues! Thanks 🙏🏼