Proper handling of ProduceFailureError?

[rjemanuele]

What is the proper handling of a ProduceFailureError? It is a very general, non-specific error.

My producer is set up :

self.kafka.topics['mytopic'].get_producer(
                delivery_reports=False,
                linger_ms=0,
                queue_empty_timeout_ms=1,
                sync=True
            )

In my code I get an exception some of the time:

  File "/usr/local/lib/python3.7/site-packages/pykafka/producer.py", line 411, in produce
    raise ProduceFailureError("Delivery report not received after timeout")
pykafka.exceptions.ProduceFailureError: Delivery report not received after timeout

What is the proper course of action to handle this? Right now, I retry the 'produce' call by that'll sometimes fail also.

Thanks

[rubinatorz]

Hi @emmett9001

I'm also getting this error on moments when one of the nodes in the cluster is down for maintenance. Consumers successfully reconnect, but the Producers keep failing with this "Delivery report not received after timeout". Those messages are not delivery at all. Once I manually restart my application, the producers are working again, but the messages that should be sent are lost. Restarting manually is not a very proper solution :) So my question is, how do I properly continue/reconnect when getting this error?

Regards, Ruben