confluentinc
Onprem SQL to AWS Confluent Cloud Kafka "Not selecting any broker for cluster connection"
Description
Trying to connect an on-prem HVR hub server to an AWS Confluent Cloud Kafka instance. With proper API username, password, and endpoint, error displayed is "Local: Broker transport failure' during fetching metadata for Kafka broker" Digging into the log files indicates that the HVR hub server uses the librdkafka library to connect.
From log file: 2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request ... y259p.us-east-1.aws.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> SSL_HANDSHAKE 2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-BROADCAST: rdkafka#producer-1: [thrd:sasl_ssl://bootstr]: Broadcasting state change 2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:app]: Cluster connection already in progress: application metadata request 2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection 2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection 2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection 2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection 2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection 2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
How to reproduce
RedHat 8.8 and v1.9.2 of library
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
- [x] librdkafka version (release number or git tag):
1.9.2 - [X] Apache Kafka version:
AWS managed service - [X] librdkafka client configuration:
not aware of any config file - [X] Operating system:
RedHat 8.8 - [X] Provide logs (with
debug=..as necessary) from librdkafka: 'not sure how' - [X] Provide broker log excerpts: 'don't have any'
- [X] Critical issue 'HVR and Confluent stuck too'
There seems to be a connectivity issue in your environment, the connection doesn't go beyond the SSL_HANDSHAKE state. Could you enable the "debug" property? Don't know how to pass it in the HVR hub server
