Confluent Cloud topic config changes cause Terraform to fail

[ayazabbas]

Hi there, I am using this provider to manage topics on Confluent Cloud. Recently, they added a new configuration value to the topics, and more recently they removed that same value, and both times it caused this error:

Error: Error waiting for topic (production.exporter.flat_submissions) to become ready: couldn't find resource (21 retries)

I believe it may be related to this issue https://github.com/Mongey/terraform-provider-kafka/issues/35

My terraform configuration:

resource "kafka_topic" "exporter_flat_submissions_topic" {
  name               = "production.exporter.flat_submissions"
  replication_factor = 3
  partitions         = 6
}

Then to fix it when the new config value was added I had to explicitly define it:

resource "kafka_topic" "exporter_flat_submissions_topic" {
  name               = "production.exporter.flat_submissions"
  replication_factor = 3
  partitions         = 6

  config = {
    "confluent.prefer.tier.fetch.ms" = "900000"
  }
}

[signaleleven]

We observed the same issue while using the provider to manage topics on multiple confluent cloud based kafka clusters. First we had to add that config ("confluent.prefer.tier.fetch.ms" = "900000") to every topic in our terraform code, to avoid terraform trying to remove it, and then we had to remove it to avoid terraform trying to add it. I think it's simply something that was added and removed on confluent's side, not really a fault of the provider here. 🤷

[muscovitebob]

Hit this today with

"compression.type"  = "producer"

which Confluent apparently removes from topics, or otherwise silently disallows. In my case I did not actually end up needing this setting but just a heads up for anyone else.