Kafka connections showing strange behaviors - not connecting, and not attempting to reconnect

[johnhtodd]

Describe the bug Strange localhost behaviors. I see (using tshark) a connection attempt to 127.0.0.1:9092 when I first start up the package, but then on repeated connection attempts there are no further packets sent to 127.0.0.1:9092.

Additionally, it shows strange port numbers and IP addresses on first attempt - see the logfile.

To Reproduce Set up kafka broker on localhost, configure system per config example below.

Expected behavior I expect a connection to my kafka broker and topic.

Additional context Kafka config file:

 - name: kafka-out
    kafkaproducer:
      remote-address: 127.0.0.1
      remote-port: 9092
      connect-timeout: 5
      retry-interval: 10
      flush-interval: 30
      tls-support: false
      tls-insecure: false
      tls-min-version: 1.2
      ca-file: ""
      cert-file: ""
      key-file: ""
      sasl-support: false
      sasl-mechanism: PLAIN
      sasl-username: "user"
      sasl-password: "kmazqAIqrx7L"
      mode: flat-json
      buffer-size: 100
      topic: "my-topic"
      partition: 0
      chan-buffer-size: 65535
      compression: "none"

Output on log:

INFO: 2024/01/01 06:39:02.051467 [dnsdist-from-outside] collector=dnstap - is listening on tcp://[::]:59312
INFO: 2024/01/01 06:39:02.051637 [prom] logger=prometheus - is listening on [::]:8082
ERROR: 2024/01/01 06:39:02.194843 [kafka-out] logger=kafka - failed to dial: failed to open connection to debian:9092: dial tcp: lookup debian on 127.0.0.53:53: server misbehaving
INFO: 2024/01/01 06:39:02.194915 [kafka-out] logger=kafka - retry to connect in 10 seconds
INFO: 2024/01/01 06:39:03.889922 [dnsdist-from-outside] collector=dnstap#1 - new connection from 127.0.0.1:59888
INFO: 2024/01/01 06:39:03.889976 [dnsdist-from-outside] processor=dnstap#1 - initialization...
INFO: 2024/01/01 06:39:03.897504 [dnsdist-from-outside] processor=dnstap#1 - waiting dns message to process...
INFO: 2024/01/01 06:39:03.899095 [dnsdist-from-outside] collector=dnstap#1 - receiver framestream initialized
ERROR: 2024/01/01 06:39:12.051153 [kafka-out] logger=kafka - failed to established connection
INFO: 2024/01/01 06:39:12.196031 [kafka-out] logger=kafka - connecting to kafka=127.0.0.1:9092 partition=0 topic=my-topic
ERROR: 2024/01/01 06:39:12.196328 [kafka-out] logger=kafka - failed to dial: failed to open connection to 127.0.0.1:9092: dial tcp 127.0.0.1:9092: operation was canceled
INFO: 2024/01/01 06:39:12.196378 [kafka-out] logger=kafka - retry to connect in 10 seconds
ERROR: 2024/01/01 06:39:22.196729 [kafka-out] logger=kafka - failed to established connection
INFO: 2024/01/01 06:39:22.196908 [kafka-out] logger=kafka - connecting to kafka=127.0.0.1:9092 partition=0 topic=my-topic
ERROR: 2024/01/01 06:39:22.197070 [kafka-out] logger=kafka - failed to dial: failed to open connection to 127.0.0.1:9092: dial tcp 127.0.0.1:9092: operation was canceled
INFO: 2024/01/01 06:39:22.197093 [kafka-out] logger=kafka - retry to connect in 10 seconds

[dmachard]

Strange localhost behaviors. I see (using tshark) a connection attempt to 127.0.0.1:9092 when I first start up the package, but then on repeated connection attempts there are no further packets sent to 127.0.0.1:9092.

Issue identified, the buffer is no more cleared after some connection attempts fix to come...