producer times out and closes

[dludwig]

Hello

My use case is for a long running system that produces messages by event triggers (think something like a web server that produces data for pulsar on HTTP events)

I find this works ok for a brief period of time, but then the producer times out and subsequent events then fail to get sent to pulsar

my work around is to open the connection, use the producer to send the data to pulsar, then close the connection every time -- this seems like a waste of time connecting/disconnecting

is there a way (or can there be a way) to make the producer connection long lasting? or auto-reconnecting?

thanks

[sijie]

@dludwig Timeout can happen if the maxPendingMessages is set to be too small. You can consider increasing the maxPendingMessages value when creating a producer.

[cavanaug]

@sijie is this because of a scaling or problem at the broker not ack()ing in time?? Or is this configuration something that needs to be scaled up as the size/volume of data increases?

Im a little nervous to just start growing something like maxPendingMessages without understanding why the max is being reached?

[sijie]

It can happen if the publish time increase (broker not ack in time). Also if the volume of traffic produced by your client increases, you need to consider increasing the setting as well. Since I don't have any visibility in your environment, it is hard for me to tell. I'd recommend you check your cluster to understand if there are any changes related to publishing latency or traffic patterns.