Kafka client keeps disconnecting after moving to SSL

Open adarsh-ns opened this issue 2 years ago • 2 comments

Description

After upgrading kafka client to connect using SSL, which gets terminated by a haproxy, we see kafka client keeps disconnecting every 10 seconds. There is no loss to functionality but it causes inconsistency in our application.

This issue looks similar to the one reported on Confluent Kafka .NET library as well (https://github.com/strimzi/strimzi-kafka-operator/discussions/6497) and was fixed. Can we have this fix in confluent-kafka-python as well, please?

How to reproduce

Enable SSL in Kafka client (producer or consumer) and try to connect through a haproxy which terminates SSL.

confluent-kafka-python version: 1.9.2 librdkafka version version:

import confluent_kafka as ck ck.version() ('1.9.2', 17367552) ck.libversion() ('1.9.2', 17367807)

Apache Kafka broker version: 2.4.0 Operating system: Ubuntu 20.04

Client configuration: {'bootstrap.servers': 'kafka01:9092', 'group.id': 'agentv2-agent01', 'auto.offset.reset': 'latest', 'enable.auto.commit': True, 'client.id': 'consumer-agent01-3595636', 'partition.assignment.strategy': 'roundrobin,range', 'conne ctions.max.idle.ms': 800000, 'max.poll.interval.ms': 600000, 'session.timeout.ms': 600000, 'security.protocol': 'ssl', 'enable.ssl.certificate.verification': False, 'ssl.endpoint.identification.algorithm': 'none'}

Client Logs

%7|1672343321.636|FAIL|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Disconnected (after 10008ms in state UP,
1 identical error(s) suppressed) (_TRANSPORT): identical to last error
%6|1672343321.636|FAIL|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Disconnected (after 10008ms in state UP,
1 identical error(s) suppressed)
%7|1672343321.636|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state UP -> DOWN
%7|1672343321.636|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state DOWN -> INIT
%7|1672343341.686|FAIL|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: kafka01:9092: Disconnected (after 30058ms in state UP, 1 ide
ntical error(s) suppressed) (_TRANSPORT): identical to last error
%6|1672343341.686|FAIL|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: kafka01:9092: Disconnected (after 30058ms in state UP, 1 ident
ical error(s) suppressed)
%7|1672343341.686|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state UP -> DOWN
%7|1672343341.686|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state DOWN -> INIT
%7|1672343341.686|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state INIT -> TRY_CONNECT
%7|1672343341.686|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: broker in state TRY_CONNECT connecting
%7|1672343341.686|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state TRY_CONNECT -> CONNECT
%7|1672343341.687|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Connecting to ipv4#198.18.29.117:9092 (ssl) with socket 13
%7|1672343341.687|REQERR|consumer-agent01-3595636#consumer-1| [thrd:main]: GroupCoordinator/1: JoinGroupRequest failed: Local: Broker transport failure: actions Retry
%7|1672343341.687|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Connected to ipv4#198.18.29.117:9092
%7|1672343341.687|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1672343341.687|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:main]: ssl://kafka01:9092/1: Selected for cluster connection: coordinator query (broker has 3 co
nnection attempt(s))
%7|1672343341.687|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Received CONNECT op
%7|1672343341.687|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state INIT -> TRY_CONNECT
%7|1672343341.687|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: broker in state TRY_CONNECT connecting
%7|1672343341.687|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state TRY_CONNECT -> CONN
ECT
%7|1672343341.687|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Connecting to ipv4#198.18.29.117:9092
(ssl) with socket 17
%7|1672343341.687|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Connected to ipv4#198.18.29.117:9092
%7|1672343341.687|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state CONNECT -> SSL_HAND
SHAKE
%7|1672343341.690|CONNECTED|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Connected (#4)
%7|1672343341.690|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY
%7|1672343341.690|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent ApiVersionRequest (v3, 101 bytes @ 0, CorrId 11)
%7|1672343341.692|CONNECTED|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Connected (#4)
%7|1672343341.692|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state SSL_HANDSHAKE -> AP
IVERSION_QUERY
%7|1672343341.692|SEND|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Sent ApiVersionRequest (v3, 101 bytes @ 0
, CorrId 10)
%7|1672343341.700|RECV|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Received ApiVersionResponse (v3, 344 byte
s, CorrId 10, rtt 7.95ms)
%7|1672343341.700|RECV|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Received ApiVersionResponse (v3, 344 bytes, CorrId 11, rtt 10.12m
s)
%7|1672343341.700|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state APIVERSION_QUERY ->
 UP
%7|1672343341.700|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state APIVERSION_QUERY -> UP
%7|1672343341.700|SEND|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Sent MetadataRequest (v4, 61 bytes @ 0, C
orrId 11)
%7|1672343341.700|SEND|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Sent FindCoordinatorRequest (v2, 100 byte
s @ 0, CorrId 12)
%7|1672343341.707|RECV|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Received MetadataResponse (v4, 64 bytes,
CorrId 11, rtt 7.26ms)
%7|1672343341.710|RECV|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Received FindCoordinatorResponse (v2, 34
bytes, CorrId 12, rtt 10.23ms)
%7|1672343341.711|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent JoinGroupRequest (v5, 327 bytes @ 0, CorrId 12)
%6|1672343344.515|FAIL|producer-agent01-3595636#producer-2| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Disconnected (after 10008ms in state UP,
3 identical error(s) suppressed)
%7|1672343351.712|FAIL|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Disconnected (after 10012ms in state UP,
1 identical error(s) suppressed) (_TRANSPORT): identical to last error
%6|1672343351.712|FAIL|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Disconnected (after 10012ms in state UP,
1 identical error(s) suppressed)
%7|1672343351.712|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state UP -> DOWN
%7|1672343351.712|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state DOWN -> INIT
%7|1672343371.760|FAIL|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: kafka01:9092: Disconnected (after 30059ms in state UP, 1 ide
ntical error(s) suppressed) (_TRANSPORT): identical to last error
%6|1672343371.760|FAIL|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: kafka01:9092: Disconnected (after 30059ms in state UP, 1 ident
ical error(s) suppressed)
%7|1672343371.760|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state UP -> DOWN
%7|1672343371.760|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state DOWN -> INIT
%7|1672343371.760|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state INIT -> TRY_CONNECT
%7|1672343371.760|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: broker in state TRY_CONNECT connecting
%7|1672343371.760|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state TRY_CONNECT -> CONNECT
%7|1672343371.760|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Connecting to ipv4#198.18.29.117:9092 (ssl) with socket 13
%7|1672343371.760|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Connected to ipv4#198.18.29.117:9092
%7|1672343371.760|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1672343371.760|REQERR|consumer-agent01-3595636#consumer-1| [thrd:main]: GroupCoordinator/1: JoinGroupRequest failed: Local: Broker transport failure: actions Retry
%7|1672343371.760|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:main]: ssl://kafka01:9092/1: Selected for cluster connection: coordinator query (broker has 4 co
nnection attempt(s))
%7|1672343371.760|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Received CONNECT op
%7|1672343371.760|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state INIT -> TRY_CONNECT
%7|1672343371.760|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: broker in state TRY_CONNECT connecting
%7|1672343371.760|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state TRY_CONNECT -> CONN
ECT
%7|1672343371.761|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Connecting to ipv4#198.18.29.117:9092
(ssl) with socket 17
%7|1672343371.761|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Connected to ipv4#198.18.29.117:9092
%7|1672343371.761|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state CONNECT -> SSL_HAND
SHAKE
%7|1672343371.763|CONNECTED|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Connected (#5)
%7|1672343371.763|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY
%7|1672343371.763|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent ApiVersionRequest (v3, 101 bytes @ 0, CorrId 13)
%7|1672343371.765|RECV|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Received ApiVersionResponse (v3, 344 bytes, CorrId 13, rtt 1.90ms
)
%7|1672343371.765|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state APIVERSION_QUERY -> UP
%7|1672343371.765|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent MetadataRequest (v4, 61 bytes @ 0, CorrId 14)
%7|1672343371.765|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent FindCoordinatorRequest (v2, 100 bytes @ 0, CorrId 15)
%7|1672343371.765|CONNECTED|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Connected (#5)
%7|1672343371.765|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state SSL_HANDSHAKE -> AP
IVERSION_QUERY
%7|1672343371.765|SEND|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Sent ApiVersionRequest (v3, 101 bytes @ 0
, CorrId 13)
%7|1672343371.765|RECV|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Received MetadataResponse (v4, 64 bytes, CorrId 14, rtt 0.64ms)
%7|1672343371.767|RECV|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Received FindCoordinatorResponse (v2, 34 bytes, CorrId 15, rtt 1.
64ms)
%7|1672343371.767|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent JoinGroupRequest (v5, 327 bytes @ 0, CorrId 16)
%7|1672343371.767|RECV|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Received ApiVersionResponse (v3, 344 byte
s, CorrId 13, rtt 1.70ms)
%7|1672343371.767|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state APIVERSION_QUERY ->
 UP
%7|1672343371.767|SEND|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Sent MetadataRequest (v4, 61 bytes @ 0, C
orrId 14)
%7|1672343371.768|RECV|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Received MetadataResponse (v4, 64 bytes,
CorrId 14, rtt 0.82ms)
%6|1672343377.515|FAIL|producer-agent01-3595636#producer-2| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Disconnected (after 10010ms in state UP,
3 identical error(s) suppressed)
%7|1672343381.771|FAIL|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Disconnected (after 10003ms in state UP,
1 identical error(s) suppressed) (_TRANSPORT): identical to last error
%6|1672343381.771|FAIL|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Disconnected (after 10003ms in state UP,
1 identical error(s) suppressed)
%7|1672343381.771|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state UP -> DOWN
%7|1672343381.771|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state DOWN -> INIT
%7|1672343401.813|FAIL|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: kafka01:9092: Disconnected (after 30048ms in state UP, 1 ide
ntical error(s) suppressed) (_TRANSPORT): identical to last error
%6|1672343401.813|FAIL|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: kafka01:9092: Disconnected (after 30048ms in state UP, 1 ident
ical error(s) suppressed)
%7|1672343401.813|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state UP -> DOWN
%7|1672343401.813|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:main]: ssl://kafka01:9092/1: Selected for cluster connection: coordinator query (broker has 5 co
nnection attempt(s))
%7|1672343401.813|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Received CONNECT op
%7|1672343401.813|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state INIT -> TRY_CONNECT
%7|1672343401.813|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: broker in state TRY_CONNECT connecting
%7|1672343401.813|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state DOWN -> INIT
%7|1672343401.813|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state INIT -> TRY_CONNECT
%7|1672343401.813|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: broker in state TRY_CONNECT connecting
%7|1672343401.813|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state TRY_CONNECT -> CONNECT
%7|1672343401.813|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state TRY_CONNECT -> CONN
ECT
%7|1672343401.813|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Connecting to ipv4#198.18.29.117:9092 (ssl) with socket 17
%7|1672343401.813|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Connecting to ipv4#198.18.29.117:9092
(ssl) with socket 13
%7|1672343401.813|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Connected to ipv4#198.18.29.117:9092
%7|1672343401.813|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Connected to ipv4#198.18.29.117:9092
%7|1672343401.813|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state CONNECT -> SSL_HAND
SHAKE
%7|1672343401.813|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1672343401.814|REQERR|consumer-agent01-3595636#consumer-1| [thrd:main]: GroupCoordinator/1: JoinGroupRequest failed: Local: Broker transport failure: actions Retry
%7|1672343401.814|CONNECT|consumer-agent01-3595636#consumer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: coordinator query
%7|1672343401.816|CONNECTED|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Connected (#6)
%7|1672343401.816|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state SSL_HANDSHAKE -> AP
IVERSION_QUERY
%7|1672343401.816|SEND|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Sent ApiVersionRequest (v3, 101 bytes @ 0
, CorrId 15)
%7|1672343401.817|CONNECTED|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Connected (#6)
%7|1672343401.817|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY
%7|1672343401.817|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent ApiVersionRequest (v3, 101 bytes @ 0, CorrId 17)
%7|1672343401.835|RECV|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Received ApiVersionResponse (v3, 344 bytes, CorrId 17, rtt 17.25m
s)
%7|1672343401.835|STATE|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state APIVERSION_QUERY -> UP
%7|1672343401.835|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent MetadataRequest (v4, 61 bytes @ 0, CorrId 18)
%7|1672343401.835|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent FindCoordinatorRequest (v2, 100 bytes @ 0, CorrId 19)
%7|1672343401.835|RECV|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Received ApiVersionResponse (v3, 344 byte
s, CorrId 15, rtt 18.79ms)
%7|1672343401.835|STATE|consumer-agent01-3595636#consumer-1| [thrd:ssl://kafka01:9092/bootstrap]: ssl://kafka01:9092/1: Broker changed state APIVERSION_QUERY ->
 UP
%7|1672343401.839|RECV|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Received MetadataResponse (v4, 64 bytes, CorrId 18, rtt 4.54ms)
%7|1672343401.841|RECV|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Received FindCoordinatorResponse (v2, 34 bytes, CorrId 19, rtt 6.
60ms)
%7|1672343401.842|SEND|consumer-agent01-3595636#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/1: Sent JoinGroupRequest (v5, 327 bytes @ 0, CorrId 20)

Broker Log Excerpts

[2023-01-06 06:17:25,953] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2023-01-06 06:21:08,406] INFO [GroupCoordinator 1]: Preparing to rebalance group agentv2_agent01 in state PreparingRebalance with old generation 0 (__consumer_offsets-14) (reason: Adding new member consumer-agent01-207166-11df1c98-8e7c-44ab-b624-655595af2737 with group instanceid None) (kafka.coordinator.group.GroupCoordinator)
[2023-01-06 06:21:11,410] INFO [GroupCoordinator 1]: Stabilized group agentv2_agent01 generation 1 (__consumer_offsets-14) (kafka.coordinator.group.GroupCoordinator)
[2023-01-06 06:21:11,414] INFO [GroupCoordinator 1]: Assignment received from leader for group agentv2_agent01 for generation 1 (kafka.coordinator.group.GroupCoordinator)
[2023-01-06 06:27:25,953] INFO [GroupMetadataManager brokerId=1] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)

Jan 06 '23 06:01 adarsh-ns

Not sure if this is your issue, but we found that enable.ssl.certificate.verification only takes a string. Can you change enable.ssl.certificate.verification: "false" and see if that fixes your issue?

Aug 28 '23 14:08 yzhan289

@adarsh-ns can you confirm if the suggestion above solves your issue?

Feb 13 '24 02:02 nhaq-confluent

confluent-kafka-python confluent-kafka-python copied to clipboard

Kafka client keeps disconnecting after moving to SSL

Description

How to reproduce

confluent-kafka-python
confluent-kafka-python copied to clipboard