Provider not working since version 0.6

[Aransh]

Been using this provider for a while now, since version 0.6 was released I am unable to use it..

I am getting this error when trying to manage "kafka_topic" resources:

│ Error: kafka: client has run out of available brokers to talk to: 3 errors occurred: │ * EOF │ * EOF │ * EOF │ │ │ with module.eclector.kafka_topic.main["eclector-waf-triggers-dlr-5"], │ on .terraform/modules/eclector/kafka_topics/main.tf line 26, in resource "kafka_topic" "main": │ 26: resource "kafka_topic" "main" { │

Reverting the version 0.5.4 (with no other configuration changes) works right away. Any directions would be greatly appreciated

[Mongey]

@Aransh what Kafka version are you using?

[Aransh]

I'm actually using RedPanda (a Kafka-compatible broker), was probably relevant to mention 😅. I tried asking their team if there's any solid answer to your question (as in, is there some internal kafka version used), this is their response:

I would just say you’re using Redpanda :slightly_smiling_face: Looking at the change log, this could either be something to do with Sarama v1.40.1 versus v1.42.1, or the Net.Proxy changes they’ve added. https://github.com/Mongey/terraform-provider-kafka/compare/v0.5.4...v0.6.0

[arkadiusjonczek]

Hi @Aransh , I have the same problem. Can you please share your solution if you found something?

[Aransh]

@arkadiusjonczek no solution yet except using the old provider version. Waiting to see if @Mongey has any possible solution

[yermulnik]

We're hitting the same issue with AWS Managed Kafka (MSK). Different versions of MSK. I'll ask my colleague to provided details.

[srikanth520]

@Mongey After upgrading to 0.6.0 version Getting this error

12:57:20 Error: kafka: client has run out of available brokers to talk to: 3 errors occurred: 12:57:20 * EOF 12:57:20 * EOF 12:57:20 * EOF 12:57:20
12:57:20
12:57:20 with kafka_topic.topics["devopstest.topic.empty2.config"], 12:57:20 on main.tf line 1, in resource "kafka_topic" "topics": 12:57:20 1: resource "kafka_topic" "topics" {

TLS Brokers AWS MSK Kafka version - 2.2.1 Terraform version: 1.5.7

Note: Same code is working for 0.5.4 version

[yermulnik]

@Mongey @errm Is this being worked on please?

Also, as it was already suggested, documentation update would be great improvement.

Thank you.

[srikanth520]

@Mongey @errm After upgrading provider version to 0.7.0 also. Still getting the error for TLS communication brokers which was occured for 0.6.0 version also.

with 0.5.4 version able to create to topics.

Error: kafka: client has run out of available brokers to talk to: 3 errors occurred:

[EliranTurgeman]

Same issue here. Someone knows what can be done??

[EliranTurgeman]

Hi Team, Any update?

[Mongey]

@EliranTurgeman / @srikanth520 / @yermulnik / @Aransh Can you please provide examples of your configuration, along with details of the Kafka service, and versions you're using.

[yermulnik]

@Mongey Please see https://github.com/Mongey/terraform-provider-kafka/issues/383#issuecomment-1976592892 — @srikanth520 and I are on the same team. Please let us know what additional details you'd like us to provide.

[Mongey]

@yermulnik There is no terraform config provided.

[yermulnik]

Ouch, I just double checked and couldn't reproduce the issue (currently we're at v0.7.1 of Kafka provider). As in I was able to connect to both Anon TLS and AWS-IAM clusters w/o issues. Even tried downgrading to v0.6.0 and still have no issues with Anon TLS cluster 🤔

I'll talk to @srikanth520 to see whether I might be missing something from his setup. Will keep you posted.

UPD: I re-read through the history of our internal ticket and it looks like the issue was with MSK cluster of 2.2.1 version, while there were no issues with clusters with version 3.5.1. Recently we finished off upgrading all of our MSK clusters to 3.5.1, so we're not anymore able to confirm the issue 🤷🏻

[Aransh]

@Mongey I can try and help with that. As I mentioned here - https://github.com/Mongey/terraform-provider-kafka/issues/383#issuecomment-1956799630, we are using redpanda (which works with this provider up to version 0.5.4).

Relevant terraform files are pretty standard: provider:

provider "kafka" {
  bootstrap_servers = ["<broker-1>:31092,<broker-2>:31092,<broker-3>:31092"]
  sasl_mechanism    = "scram-sha512"
  sasl_username     = <username>
  sasl_password     = <password>
}

example topic:

  resource "kafka_topic" "main" {
      config  = {
          "cleanup.policy"    = "delete"
          "max.message.bytes" = "2097164"
          "retention.ms"      = "86400000"
          "segment.ms"        = "82800000"
      }
      name               = "<topic-name>"
      partitions         = 6
      replication_factor = 3
  }

And on apply, after approving the plan:

kafka_topic.main["<topic-name>"]: Creating...
kafka_topic.main["<topic-name>"]: Still creating... [10s elapsed]
╷
│ Error: kafka: client has run out of available brokers to talk to: 3 errors occurred:
│ 	* EOF
│ 	* EOF
│ 	* EOF

[Mongey]

@Aransh I have just tried with red panda serverless and cannot reproduce.

[Aransh]

@Mongey Hmmm... that's odd, we've had this issue reproduced by 3 of our teams in several different redpanda deployments... We are using self-hosted redpanda, does serverless tell you the redpanda version? (our main environment uses v24.1.1)

[tommyminds]

I'm having the same issue with my self-hosted redpanda broker. In the redpanda logs I see the following error:

kafka - connection_context.cc:362 - Error while processing request from 10.244.0.84:55402 - Unsupported version 9 for metadata API.

Maybe that is related?

[Mongey]

I'm having the same issue with my self-hosted redpanda broker. In the redpanda logs I see the following error:

kafka - connection_context.cc:362 - Error while processing request from 10.244.0.84:55402 - Unsupported version 9 for metadata API.

Maybe that is related?

Maybe try setting kafka_version in the provider config to something lower than the default (2.7.0)

[Aransh]

Maybe try setting kafka_version in the provider config to something lower than the default (2.7.0)

Already tried that, no dice...

[voutilad]

kafka_version = "2.1.0" works for me with Redpanda 24.1.

[Aransh]

kafka_version = "2.1.0" works for me with Redpanda 24.1.

Can confirm specifically setting this does indeed work, can probably close this then. Thanks @voutilad !