Debugging sasl_plaintext... Timed out 1 in-flight

[zbs]

We have producers that occasionally log this error after they've stopped sending messages to the broker

2025-09-23 19:09:45.993482823: WARN:          KafkaProducer.cpp(52): Kafka: [thrd:sasl_plaintext://7.9.213.19:9092/0]: sasl_plaintext://7.9.213.19:9092/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
2025-09-23 19:09:45.993535068: ERRR:          KafkaProducer.cpp(48): Kafka: [thrd:sasl_plaintext://7.9.213.19:9092/0]: sasl_plaintext://7.9.213.19:9092/0: 1 request(s) timed out: disconnect (average rtt 7197.479ms) (after 5362051ms in state
2025-09-23 19:09:47.318182751: WARN:          KafkaProducer.cpp(52): Kafka: [thrd:sasl_plaintext://7.9.213.19:9092/0]: sasl_plaintext://7.9.213.19:9092/0: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 1 partially-sent requests

This error usually happens after the producer has successfully sent some messages. Then these errors will occur, messages will fail to send, and sometimes the producers successfully reconnect later.

Because these errors are non-deterministic, they are quite difficult to troubleshoot. Can anyone indicate which rabbit hole I should start to go down? Is this an error with authentication? Is it a red herring and it's a producer performance issue? Is this typically masking a network issue?

Note I'm using version 1.7.