Disconnected while requesting ApiVersion - Kubernetes Only

[MacherelR]

Description

I'm facing a problem since I deployed redpanda inside a kubernetes cluster (following those instructions: https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/kubernetes/kubernetes-deploy/?tab=tabs-1-helm#tabs-1-helm) :

redpanda.lysredge.svc.cluster.local:9093/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 13ms in state APIVERSION_QUERY, 3 identical error(s) suppressed)

I was previously testing with the redpanda docker compose version locally on my mac and never encountered this problem... It only appears when deploying redpanda in kubernetes (in my case a k3s cluster).

How to reproduce

• setup basic k3s cluster
• deploy redpanda within (https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/kubernetes/local-guide/#:~:text=%3D%3ANoSchedule-,Deploy%20Redpanda%20and%20Redpanda%20Console,-In%20this%20step)
• Create a kafka producer with config {"bootstrap.servers": "redpanda..svc.cluster.local:9093"}

Checklist

Please provide the following information:

• [ X] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): confluent_kafka : ('2.2.0', 33685504) || librdkafka: ('2.2.0', 33685759)
• [ X] Apache Kafka broker version: output of rpk version command : v23.2.14 (rev b5a8b9a28)
• [ X] Client configuration: {"bootstrap.servers": "redpanda..svc.cluster.local:9093"}
• [ X] Operating system: Kubernetes / python image
• [ X] Provide client logs (with 'debug': '..' as necessary) : %6|1700211621.386|FAIL|rdkafka#producer-1| [thrd:redpanda..svc.cluster.local:9093/bootstrap]: redpanda..svc.cluster.local:9093/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 13ms in state APIVERSION_QUERY, 3 identical error(s) suppressed)

[jozeecp]

Any chance you've found an answer to this issue? I'm seeing the same thing.

[MacherelR]

Hi @jozeecp, Not sure anymore how I worked around this but if I remember well, it was an issue with TLS being enabled in the default values of redpanda deployment. You got in redpanda values a key called tls enable which should be set to false. In my case I didn't need TLS as there were no external access to redpanda so I was able to disable it. Please let me know if that fixes your problem, and if not I might try to help you debugging it !