Inability to NOT use files in provider

[brandonpalmer]

I'm able to connect to my provider using:

provider "kafka" {
  bootstrap_servers = split(",", data.aws_msk_cluster.msk_cluster_1.bootstrap_brokers_tls)
  ca_cert           = file("ca.pem")
  client_cert       = file("cert.pem")
  client_key        = file("key.d.pem")
  tls_enabled       = true
}

But when I try to get the CA / client cert / client key NOT from a file, I am getting an error:

provider "kafka" {
  bootstrap_servers = split(",", data.aws_msk_cluster.msk_cluster_1.bootstrap_brokers_tls)
  ca_cert           = "-----BEGIN CERTIFICATE----- MII..... [removed] ....HXw== -----END CERTIFICATE-----"
  client_cert       = file("cert.pem")
  client_key        = file("key.d.pem")
  tls_enabled       = true
}

Error: open -----BEGIN CERTIFICATE----- MII..... [removed] ....HXw== -----END CERTIFICATE-----: no such file or directory

Per the docs: The CA certificate or path to a CA certificate file to validate the server's certificate.

[brandonpalmer]

Best I can guess, there is somehow an issue with processing the PEM when something other than file() is used?

https://github.com/Mongey/terraform-provider-kafka/blob/master/kafka/config.go#L86

[Mongey]

Seems like a bug to me. Have you got an example of your whole config, and I can spin up an AWS MSK cluster and see what's going on.

I think with the second config is that the ca_cert is possible not being read correctly due to how it's formatted. Maybe you could try using HEREDOCs to insert the certificate.

Like so

provider "kafka" {
  bootstrap_servers = split(",", data.aws_msk_cluster.msk_cluster_1.bootstrap_brokers_tls)
  ca_cert           = <<CA
-----BEGIN CERTIFICATE-----
MIIblahblahblah==
-----END CERTIFICATE-----
CA

  client_cert       = file("cert.pem")
  client_key        = file("key.d.pem")
  tls_enabled       = true
}

[brandonpalmer]

I was able to come to a work-around by doing a b64encode of the pem file and then base64decode that as the input for ca_cert. HEREDOC wouldn't work for me because the PEM data is actually stores in AWS Secrets.

data "aws_secretsmanager_secret_version" "msk_core1_master" {
  secret_id = "msk/msk-core1"
}

provider "kafka" {
  bootstrap_servers = split(",", data.aws_msk_cluster.msk_cluster_1.bootstrap_brokers_tls)
  ca_cert           = base64decode(jsondecode(data.aws_secretsmanager_secret_version.msk_core1_master.secret_string)["ca"])
  client_cert       = base64decode(jsondecode(data.aws_secretsmanager_secret_version.msk_core1_master.secret_string)["client_cert"])
  client_key        = base64decode(jsondecode(data.aws_secretsmanager_secret_version.msk_core1_master.secret_string)["client_key"])
  tls_enabled       = true
}

It seems that the go pem.Decode() isn't tolerant of string formats (at least far less than the openssl tools).

I suspect updating the docs to say either use HEREDOCs or base64 encode it would be helpful (would have saved me many hours of research at least).