Producer logging thousands of errors in a very short timespan (Broker transport failure)

Open Atheuz opened this issue 2 years ago • 8 comments

Description

I have a problem with a producer in an API. Specifically, once in a while it'll throw errors like SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN), which wouldn't be a problem on its own if it was a few times, but the problem is that whenever it happens that there's a transport failure I will get THOUSANDS of log entries, like it's retrying it every single millisecond.

For instance, here's 1½ minute where it logged that error 10 000 times:

I'm fine with it logging errors and retrying the request, but I don't understand why it's retrying it so frequently. I've tried fiddling with producer config to make it back off, but even with the defaults it seems like it shouldn't really be doing it that often.

Here's an example of the error:

FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker:9092/bootstrap]: sasl_ssl://broker:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)

My producing code looks like this:

def produce_event(
    topic, event_data, kafka_producer
):
    """Produce an Event to Kafka."""
    event_id = str(uuid.uuid4())
    event_dt = datetime.datetime.utcnow()
    event = Event(
        event_id=UUID(value=event_id),
        event_datetime=Timestamp(seconds=int(event_dt.timestamp()), nanos=0),
        event_data=JsonString(content=json.dumps(event_data)),
    )
    kafka_producer.produce(topic, key=event_id, value=event, on_delivery=kafka_delivery_report)
    kafka_producer.poll()

How to reproduce

Unknown

Checklist

Please provide the following information:

[x] confluent-kafka-python and librdkafka version: ('1.8.2', 17302016) and ('1.8.2', 17302271)
[x] Apache Kafka broker version: 2.7.0 (on Confluent Platform 6.1.4)
[x] Client configuration: { "retry.backoff.ms": 1000, "reconnect.backoff.ms": 500, "reconnect.backoff.max.ms": 5000, "key.serializer": string_serializer, "value.serializer": proto_serializer }
[x] Operating system: Debian
[x] Provide client logs (with 'debug': '..' as necessary)
[ ] Provide broker log excerpts
[ ] Critical issue

Apr 25 '22 10:04 Atheuz

Can you provide an excerpt of the logs, e.g., 100 lines or so?

Apr 25 '22 10:04 edenhill

Here's a few, is this what you meant?

"@timestamp","_id","_index","error.message"
"Apr 23, 2022 @ 13:25:54.126",aQOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.126",ZwOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.125",ZQOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.125",YwOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.125",YQOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.124",XwOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.124",XQOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.124",WwOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.123",WQOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.123",VwOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.123",VQOZVoABI6pBUPGKsRFO,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.122",7wOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 18ms in state CONNECT, 37 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.122",7gOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.122",7AOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.121",6gOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.121",6AOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.121",5gOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.120",5AOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.120",4gOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.120",4AOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.119",3gOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.119",3AOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.119",2gOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.118",2AOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.117",1gOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 8ms in state CONNECT, 35 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.117",1QOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.117",0wOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.117",0QOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.116",zwOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.116",zQOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.115",ywOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.115",yQOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.115",xwOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.114",xQOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.114",wwOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.114",wQOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.114",vwOZVoABI6pBUPGKsRAP,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.113",WAOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.113",VgOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 7ms in state CONNECT, 38 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.113",VQOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.113",UwOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.113",UQOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.112",TwOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.112",TQOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.112",SwOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.112",SQOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.111",RwOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.111",RQOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.111",QwOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.111",QQOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.110",PwOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.110",PQOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 20ms in state CONNECT, 36 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.110",PAOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.110",OgOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.109",OAOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.109",NgOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.109",NAOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.109",MgOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.108",MAOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.108",LgOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.108",LAOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.108",KgOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.107",KAOZVoABI6pBUPGKsBDf,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.107","7AOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.107","6gOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.107","6AOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 8ms in state CONNECT, 34 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.107","5wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.106","5QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.046","4wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.046","4QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.046","3wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.045","3QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.045","2wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.045","2QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.045","1wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.044","1QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.044","0wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.044","0QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.044","zwOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 2ms in state CONNECT, 38 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.043","zgOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.043","zAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.043","ygOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.043","yAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.042","xgOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.042","xAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.042","wgOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.042","wAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.041","vgOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.041","vAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.041",VgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.041",VAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.040",UgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 2ms in state CONNECT, 32 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.040",UQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.040",TwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.040",TQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.040",SwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.039",SQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.039",RwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.039",RQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.039",QwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.038",QQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.038",PwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.038",PQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.038",OwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.037",OQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 1ms in state CONNECT, 39 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.037",OAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.037",NgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.037",NAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.037",MgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.036",MAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.036",LgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.036",LAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.036",KgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.035",KAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.035",JgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.035",FgOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.035",FAOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.034",EgOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 6ms in state CONNECT, 39 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.034",EQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.034",DwOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.034",DQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.033",CwOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.033",CQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.033",BwOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.033",BQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.032",AwOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.032",AQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.032","_wOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.032","_QOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.031","-wOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.031","-QOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.031",9wOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.031",9QOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 7ms in state CONNECT, 41 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.031",9AOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.030",8gOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.030",8AOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.030",7gOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.030",7AOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.029",6gOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.029",6AOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.029",5gOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.029",PwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.028",PQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.028",OwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.028",OQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.027",NwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 10ms in state CONNECT, 33 identical error(s) suppressed)"
"Apr 23, 2022 @ 13:25:54.027",NgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.027",NAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.027",MgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.027",MAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.026",LgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.026",LAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.026",KgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.026",KAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.025",JgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.025",JAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.025",IgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.024",IAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.024",HgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 7ms in state CONNECT)"
"Apr 23, 2022 @ 13:25:54.024",HQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 23, 2022 @ 13:25:54.024",GwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"

Apr 25 '22 10:04 Atheuz

Here's a different set with a few more fields, and not just limited to error logs:

"@timestamp","log.logger","log.level","_id","_index",message,"log.origin.file.name","log.origin.file.line","process.thread.name"
"Apr 23, 2022 @ 13:25:54.046","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"5AOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 67ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.046","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"4wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.046","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"4gOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 140ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.046","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"4QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.046","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"4AOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 66ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.046","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"3wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.045","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"3gOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 70ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.045","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"3QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.045","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"3AOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 80ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.045","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"2wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.045","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"2gOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 54ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.045","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"2QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.045","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"2AOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 80ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.045","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"1wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.044","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"1gOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 84ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.044","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"1QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.044","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"1AOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 85ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.044","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"0wOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.044","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"0gOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 80ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.044","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"0QOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.044","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"0AOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 189ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.044","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"zwOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 2ms in state CONNECT, 38 identical error(s) suppressed)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.043","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"zgOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.043","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"zQOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 54ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.043","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"zAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.043","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"ywOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 65ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.043","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"ygOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.043","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"yQOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 73ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.043","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"yAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.043","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"xwOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 50ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.042","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"xgOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.042","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"xQOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 70ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.042","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"xAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.042","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"wwOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 65ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.042","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"wgOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.042","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"wQOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 94ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.042","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"wAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.042","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"vwOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 71ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.041","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"vgOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.041","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"vQOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 103ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.041","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"vAOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.041","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"uwOZVoABI6pBUPGKsA-r","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 67ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.041","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,VgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.041","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,VQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 169ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.041","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,VAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.041","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,UwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 79ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.040","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,UgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 2ms in state CONNECT, 32 identical error(s) suppressed)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.040","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,UQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.040","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,UAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 81ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.040","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,TwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.040","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,TgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 52ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.040","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,TQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.040","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,TAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 70ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.040","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,SwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.040","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,SgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 89ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.039","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,SQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.039","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,SAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 70ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.039","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,RwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.039","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,RgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 53ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.039","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,RQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.039","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,RAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 80ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.039","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,QwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.039","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,QgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 51ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.038","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,QQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.038","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,QAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 66ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.038","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,PwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.038","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,PgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 68ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.038","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,PQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.038","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,PAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 106ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.038","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,OwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.038","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,OgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 82ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.037","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,OQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 1ms in state CONNECT, 39 identical error(s) suppressed)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.037","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,OAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.037","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,NwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 81ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.037","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,NgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.037","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,NQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 82ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.037","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,NAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.037","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,MwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 181ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.037","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,MgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.036","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,MQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 67ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.036","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,MAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.036","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,LwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 70ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.036","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,LgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.036","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,LQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 95ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.036","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,LAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.036","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,KwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 157ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.036","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,KgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.035","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,KQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 65ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.035","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,KAOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.035","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,JwOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 54ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.035","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,JgOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.035","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,JQOZVoABI6pBUPGKsA9p,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 83ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.035","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,FgOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.035","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,FQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 123ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.035","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,FAOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.034","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,EwOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 120ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.034","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,EgOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 6ms in state CONNECT, 39 identical error(s) suppressed)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.034","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,EQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.034","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,EAOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 69ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.034","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,DwOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.034","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,DgOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 67ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.034","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,DQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.034","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,DAOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 107ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.033","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,CwOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.033","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,CgOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 90ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.033","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,CQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.033","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,CAOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 92ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.033","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,BwOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.033","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,BgOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 86ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.033","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,BQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.033","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,BAOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 94ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.032","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,AwOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.032","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,AgOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 80ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.032","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,AQOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.032","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,AAOZVoABI6pBUPGKsA4E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 69ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.032","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"_wOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.032","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"_gOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 63ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.032","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"_QOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.032","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"_AOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 144ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.031","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"-wOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.031","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"-gOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 139ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.031","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"-QOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.031","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"-AOZVoABI6pBUPGKsA0E","k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 84ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.031","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,9wOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.031","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,9gOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 50ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.031","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,9QOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 7ms in state CONNECT, 41 identical error(s) suppressed)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.031","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,9AOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.030","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,8wOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 200ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.030","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,8gOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.030","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,8QOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 117ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.030","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,8AOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.030","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,7wOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 81ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.030","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,7gOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.030","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,7QOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 65ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.030","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,7AOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.029","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,6wOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 65ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.029","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,6gOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.029","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,6QOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 66ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.029","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,6AOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.029","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,5wOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 84ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.029","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,5gOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.029","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,5QOZVoABI6pBUPGKsA0E,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 53ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.029","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,PwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.028","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,PgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 76ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.028","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,PQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.028","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,PAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 64ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.028","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,OwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.028","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,OgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 101ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.028","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,OQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.028","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,OAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 52ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.027","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,NwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 10ms in state CONNECT, 33 identical error(s) suppressed)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.027","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,NgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.027","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,NQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 66ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.027","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,NAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.027","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,MwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 76ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.027","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,MgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.027","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,MQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 84ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.027","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,MAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.026","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,LwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 87ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.026","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,LgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.026","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,LQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 68ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.026","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,LAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.026","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,KwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 82ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.026","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,KgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.026","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,KQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 69ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.026","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,KAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.025","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,JwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 67ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.025","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,JgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.025","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,JQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 65ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.025","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,JAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.025","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,IwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 64ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.025","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,IgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.025","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,IQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 65ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.024","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,IAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.024","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,HwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 78ms in state AUTH_REQ)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.024","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,HgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connect to ipv4#1.1.1.1:9092 failed: Connection refused (after 7ms in state CONNECT)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.024","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,HQOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.024","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,HAOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 125ms in state AUTH_REQ, 1 identical error(s) suppressed)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.024","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,GwOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)","kafka.py",27,"Trio worker thread 1212"
"Apr 23, 2022 @ 13:25:54.024","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,GgOZVoABI6pBUPGKrw3V,"k8s-7.16.3-2022.w17","FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 181ms in state AUTH_REQ, 1 identical error(s) suppressed)","kafka.py",27,"Trio worker thread 1212"

Apr 25 '22 11:04 Atheuz

Thank you.

It is weird that the reconnect.backoff.. is not seeming to function at all.

Could you reproduce this with debug conf set to broker and provide some logs?

Thanks

Apr 25 '22 16:04 edenhill

@edenhill here:

"@timestamp","log.logger","log.level",message
"Apr 28, 2022 @ 09:08:14.237","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state INIT -> TRY_CONNECT"
"Apr 28, 2022 @ 09:08:14.237","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"RECONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Delaying next reconnect by 9809ms"
"Apr 28, 2022 @ 09:08:14.237","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.237","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 984ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.237","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.237","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 979ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.236","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 497ms in state AUTH_REQ)"
"Apr 28, 2022 @ 09:08:14.236","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state AUTH_REQ -> DOWN"
"Apr 28, 2022 @ 09:08:14.236","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN) (_TRANSPORT)"
"Apr 28, 2022 @ 09:08:14.236","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 28, 2022 @ 09:08:14.236","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: Not selecting any broker for cluster connection: still suppressed for 184ms: broker down"
"Apr 28, 2022 @ 09:08:14.236","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state DOWN -> INIT"
"Apr 28, 2022 @ 09:08:14.235","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Initializing SASL client: service name kafka, hostname broker3, mechanisms OAUTHBEARER, provider OAUTHBEARER (builtin)"
"Apr 28, 2022 @ 09:08:14.235","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.235","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"OAUTHBEARER [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL OAUTHBEARER client in state client-first-message"
"Apr 28, 2022 @ 09:08:14.235","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 885ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.235","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"OAUTHBEARER [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Built client first message"
"Apr 28, 2022 @ 09:08:14.235","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Send SASL Kafka frame to broker (1314 bytes)"
"Apr 28, 2022 @ 09:08:14.235","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 497ms in state AUTH_REQ) (_TRANSPORT)"
"Apr 28, 2022 @ 09:08:14.234","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY"
"Apr 28, 2022 @ 09:08:14.234","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"AUTH [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Auth in state APIVERSION_QUERY (handshake supported)"
"Apr 28, 2022 @ 09:08:14.234","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state APIVERSION_QUERY -> AUTH_HANDSHAKE"
"Apr 28, 2022 @ 09:08:14.234","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASLMECHS [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker supported SASL mechanisms: OAUTHBEARER"
"Apr 28, 2022 @ 09:08:14.234","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"AUTH [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Auth in state AUTH_HANDSHAKE (handshake supported)"
"Apr 28, 2022 @ 09:08:14.234","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state AUTH_HANDSHAKE -> AUTH_REQ"
"Apr 28, 2022 @ 09:08:14.233","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state TRY_CONNECT -> CONNECT"
"Apr 28, 2022 @ 09:08:14.233","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Connecting to ipv4#10.134.3.12:9092 (sasl_ssl) with socket 28"
"Apr 28, 2022 @ 09:08:14.233","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Connected to ipv4#10.134.3.12:9092"
"Apr 28, 2022 @ 09:08:14.233","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state CONNECT -> SSL_HANDSHAKE"
"Apr 28, 2022 @ 09:08:14.233","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SSLVERIFY [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker SSL certificate verified"
"Apr 28, 2022 @ 09:08:14.233","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECTED [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Connected (#1700)"
"Apr 28, 2022 @ 09:08:14.232","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: Cluster connection already in progress: broker down"
"Apr 28, 2022 @ 09:08:14.232","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state DOWN -> INIT"
"Apr 28, 2022 @ 09:08:14.232","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state INIT -> TRY_CONNECT"
"Apr 28, 2022 @ 09:08:14.232","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"RECONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Delaying next reconnect by 9811ms"
"Apr 28, 2022 @ 09:08:14.232","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 716ms: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.232","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 606ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.232","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: broker in state TRY_CONNECT connecting"
"Apr 28, 2022 @ 09:08:14.231","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"OAUTHBEARER [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Built client first message"
"Apr 28, 2022 @ 09:08:14.231","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Send SASL Kafka frame to broker (1314 bytes)"
"Apr 28, 2022 @ 09:08:14.231","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 351ms in state AUTH_REQ) (_TRANSPORT)"
"Apr 28, 2022 @ 09:08:14.231","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Disconnected (after 351ms in state AUTH_REQ)"
"Apr 28, 2022 @ 09:08:14.231","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state AUTH_REQ -> DOWN"
"Apr 28, 2022 @ 09:08:14.231","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN) (_TRANSPORT)"
"Apr 28, 2022 @ 09:08:14.231","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 28, 2022 @ 09:08:14.230","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.230","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASLMECHS [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker supported SASL mechanisms: OAUTHBEARER"
"Apr 28, 2022 @ 09:08:14.230","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 902ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.230","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"AUTH [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Auth in state AUTH_HANDSHAKE (handshake supported)"
"Apr 28, 2022 @ 09:08:14.230","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state AUTH_HANDSHAKE -> AUTH_REQ"
"Apr 28, 2022 @ 09:08:14.230","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Initializing SASL client: service name kafka, hostname broker1, mechanisms OAUTHBEARER, provider OAUTHBEARER (builtin)"
"Apr 28, 2022 @ 09:08:14.230","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"OAUTHBEARER [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL OAUTHBEARER client in state client-first-message"
"Apr 28, 2022 @ 09:08:14.229","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Connected to ipv4#10.134.3.10:9092"
"Apr 28, 2022 @ 09:08:14.229","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state CONNECT -> SSL_HANDSHAKE"
"Apr 28, 2022 @ 09:08:14.229","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SSLVERIFY [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker SSL certificate verified"
"Apr 28, 2022 @ 09:08:14.229","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECTED [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Connected (#1823)"
"Apr 28, 2022 @ 09:08:14.229","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY"
"Apr 28, 2022 @ 09:08:14.229","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"AUTH [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Auth in state APIVERSION_QUERY (handshake supported)"
"Apr 28, 2022 @ 09:08:14.229","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state APIVERSION_QUERY -> AUTH_HANDSHAKE"
"Apr 28, 2022 @ 09:08:14.228","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker changed state INIT -> TRY_CONNECT"
"Apr 28, 2022 @ 09:08:14.228","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"RECONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Delaying next reconnect by 8122ms"
"Apr 28, 2022 @ 09:08:14.228","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.228","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 979ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.228","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: broker in state TRY_CONNECT connecting"
"Apr 28, 2022 @ 09:08:14.228","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state TRY_CONNECT -> CONNECT"
"Apr 28, 2022 @ 09:08:14.228","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Connecting to ipv4#10.134.3.10:9092 (sasl_ssl) with socket 29"
"Apr 28, 2022 @ 09:08:14.227","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Disconnected (after 563ms in state AUTH_REQ) (_TRANSPORT)"
"Apr 28, 2022 @ 09:08:14.227","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Disconnected (after 563ms in state AUTH_REQ)"
"Apr 28, 2022 @ 09:08:14.227","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker changed state AUTH_REQ -> DOWN"
"Apr 28, 2022 @ 09:08:14.227","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN) (_TRANSPORT)"
"Apr 28, 2022 @ 09:08:14.227","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 28, 2022 @ 09:08:14.227","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: Not selecting any broker for cluster connection: still suppressed for 106ms: broker down"
"Apr 28, 2022 @ 09:08:14.227","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker changed state DOWN -> INIT"
"Apr 28, 2022 @ 09:08:14.226","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"AUTH [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Auth in state AUTH_HANDSHAKE (handshake supported)"
"Apr 28, 2022 @ 09:08:14.226","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker changed state AUTH_HANDSHAKE -> AUTH_REQ"
"Apr 28, 2022 @ 09:08:14.226","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Initializing SASL client: service name kafka, hostname broker2, mechanisms OAUTHBEARER, provider OAUTHBEARER (builtin)"
"Apr 28, 2022 @ 09:08:14.226","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.226","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"OAUTHBEARER [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: SASL OAUTHBEARER client in state client-first-message"
"Apr 28, 2022 @ 09:08:14.226","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 891ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.226","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"OAUTHBEARER [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Built client first message"
"Apr 28, 2022 @ 09:08:14.226","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASL [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Send SASL Kafka frame to broker (1314 bytes)"
"Apr 28, 2022 @ 09:08:14.225","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker changed state CONNECT -> SSL_HANDSHAKE"
"Apr 28, 2022 @ 09:08:14.225","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SSLVERIFY [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker SSL certificate verified"
"Apr 28, 2022 @ 09:08:14.225","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECTED [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connected (#1700)"
"Apr 28, 2022 @ 09:08:14.225","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY"
"Apr 28, 2022 @ 09:08:14.225","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"AUTH [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Auth in state APIVERSION_QUERY (handshake supported)"
"Apr 28, 2022 @ 09:08:14.225","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker changed state APIVERSION_QUERY -> AUTH_HANDSHAKE"
"Apr 28, 2022 @ 09:08:14.225","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASLMECHS [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker supported SASL mechanisms: OAUTHBEARER"
"Apr 28, 2022 @ 09:08:14.224","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 983ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.224","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.224","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 985ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.224","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: broker in state TRY_CONNECT connecting"
"Apr 28, 2022 @ 09:08:14.224","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Broker changed state TRY_CONNECT -> CONNECT"
"Apr 28, 2022 @ 09:08:14.224","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connecting to ipv4#10.134.3.11:9092 (sasl_ssl) with socket 29"
"Apr 28, 2022 @ 09:08:14.224","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker2:9092/bootstrap]: sasl_ssl://broker2:9092/2: Connected to ipv4#10.134.3.11:9092"
"Apr 28, 2022 @ 09:08:14.223","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"RECONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Delaying next reconnect by 9806ms"
"Apr 28, 2022 @ 09:08:14.223","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 712ms: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.223","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 603ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.223","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.223","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 985ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.223","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.223","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 984ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.223","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.222","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 389ms in state AUTH_REQ)"
"Apr 28, 2022 @ 09:08:14.222","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state AUTH_REQ -> DOWN"
"Apr 28, 2022 @ 09:08:14.222","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN) (_TRANSPORT)"
"Apr 28, 2022 @ 09:08:14.222","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 28, 2022 @ 09:08:14.222","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: Cluster connection already in progress: broker down"
"Apr 28, 2022 @ 09:08:14.222","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state DOWN -> INIT"
"Apr 28, 2022 @ 09:08:14.222","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state INIT -> TRY_CONNECT"
"Apr 28, 2022 @ 09:08:14.221","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 901ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.221","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state AUTH_HANDSHAKE -> AUTH_REQ"
"Apr 28, 2022 @ 09:08:14.221","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Initializing SASL client: service name kafka, hostname broker3, mechanisms OAUTHBEARER, provider OAUTHBEARER (builtin)"
"Apr 28, 2022 @ 09:08:14.221","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"OAUTHBEARER [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: SASL OAUTHBEARER client in state client-first-message"
"Apr 28, 2022 @ 09:08:14.221","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"OAUTHBEARER [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Built client first message"
"Apr 28, 2022 @ 09:08:14.221","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Send SASL Kafka frame to broker (1314 bytes)"
"Apr 28, 2022 @ 09:08:14.221","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 389ms in state AUTH_REQ) (_TRANSPORT)"
"Apr 28, 2022 @ 09:08:14.220","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECTED [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Connected (#1699)"
"Apr 28, 2022 @ 09:08:14.220","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY"
"Apr 28, 2022 @ 09:08:14.220","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"AUTH [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Auth in state APIVERSION_QUERY (handshake supported)"
"Apr 28, 2022 @ 09:08:14.220","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state APIVERSION_QUERY -> AUTH_HANDSHAKE"
"Apr 28, 2022 @ 09:08:14.220","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SASLMECHS [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker supported SASL mechanisms: OAUTHBEARER"
"Apr 28, 2022 @ 09:08:14.220","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"
"Apr 28, 2022 @ 09:08:14.220","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"AUTH [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Auth in state AUTH_HANDSHAKE (handshake supported)"
"Apr 28, 2022 @ 09:08:14.219","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 984ms: no cluster connection"
"Apr 28, 2022 @ 09:08:14.219","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: broker in state TRY_CONNECT connecting"
"Apr 28, 2022 @ 09:08:14.219","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state TRY_CONNECT -> CONNECT"
"Apr 28, 2022 @ 09:08:14.219","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Connecting to ipv4#10.134.3.12:9092 (sasl_ssl) with socket 28"
"Apr 28, 2022 @ 09:08:14.219","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Connected to ipv4#10.134.3.12:9092"
"Apr 28, 2022 @ 09:08:14.219","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state CONNECT -> SSL_HANDSHAKE"
"Apr 28, 2022 @ 09:08:14.219","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"SSLVERIFY [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker SSL certificate verified"
"Apr 28, 2022 @ 09:08:14.218","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Error,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: SASL authentication error: SaslAuthenticateRequest failed: Local: Broker transport failure (after 0ms in state DOWN)"
"Apr 28, 2022 @ 09:08:14.218","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: Not selecting any broker for cluster connection: still suppressed for 415ms: broker down"
"Apr 28, 2022 @ 09:08:14.218","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state DOWN -> INIT"
"Apr 28, 2022 @ 09:08:14.218","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Broker changed state INIT -> TRY_CONNECT"
"Apr 28, 2022 @ 09:08:14.218","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"RECONNECT [rdkafka#producer-1] [thrd:sasl_ssl://broker1:9092/bootstrap]: sasl_ssl://broker1:9092/1: Delaying next reconnect by 9803ms"
"Apr 28, 2022 @ 09:08:14.218","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"CONNECT [rdkafka#producer-1] [thrd:main]: Cluster connection already in progress: refresh unavailable topics"

Apr 28 '22 10:04 Atheuz

My "solution" to this problem is to raise the log level to CRITICAL for that specific logger, because it doesn't seem to actually impact anything.

I saw it again today, on another application, where I forgot how I solved it last time, which was as mentioned to raise the log level to critical.

I don't understand why it's not respecting reconnect.backoff.ms or retry.backoff.ms, when they're set to thousands of ms. The defaults for those configs is 100ms and it seems to be trying every ms, so even the defaults are being ignored.

EDIT: Another solution is to add a logging Filter to drop this log record.

May 09 '23 09:05 Atheuz

Hello @Atheuz, sorry for replying after a long time. I've tried to reproduce it but I couldn't and I was wondering how often do you call poll in your python application, because timestamps are generated there for all the queued log entries, so it can be that the timestamps we're seeing don't correspond to when the message was generated but to when it was logged (there is a queue for them). For example I see other discrepancies in timing, such as:

"Apr 28, 2022 @ 09:08:14.221","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Debug,"STATE [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Broker changed state AUTH_HANDSHAKE -> AUTH_REQ"
"Apr 28, 2022 @ 09:08:14.222","data_access_plugins.kafka_connectors.kafka_connector_abstractions",Info,"FAIL [rdkafka#producer-1] [thrd:sasl_ssl://broker3:9092/bootstrap]: sasl_ssl://broker3:9092/3: Disconnected (after 389ms in state AUTH_REQ)"

Can you reproduce it while polling in a loop with a timeout, without sleeps in between, like in this example?

Aug 22 '23 07:08 emasab

@emasab I am seeing the same issue using confluent-kafka-go library. Very unusual behavior which does not impact anything, but the log is cumbersome. On my machine it shows few times, on colleagues machine it shows multiple times.

Nov 28 '23 14:11 lsimunec

confluent-kafka-python confluent-kafka-python copied to clipboard

Producer logging thousands of errors in a very short timespan (Broker transport failure)

Description

How to reproduce

Checklist

confluent-kafka-python
confluent-kafka-python copied to clipboard