Flush on terminate event

[doranych]

I think, that we could wait too much, if kafka cluster is lagging.

Imagine, you have to sent 10 events to kafka. Each one will wait max 10s. In case of max waiting per message, you will get 100 sec await per process.

Sounds crazy.

What do you think about subscription on kernel.teminate event to flush?

We can call flush in cycle, to prevent message loss. In worst case process will wait 10 seconds, because of rdkafka batch collection.

[rdotter]

@doranych it could even be that the kafka message is sent via an async message, like first SQS and after that Kafka so a retry mechanism is in place. This will also prevent the wait in the interface for 10 seconds. So then the current implementation is just fine.

So it would be nice if this enhancement is been made, it should be customizable when the flush occures.

[doranych]

чт, 7 мая 2020 г., 14:43 Rudy den Otter [email protected]:

@doranych https://github.com/doranych it could even be that the kafka message is sent via an async message, like first SQS and after that Kafka so a retry mechanism is in place. This will also prevent the wait in the interface for 10 seconds. So then the current implementation is just fine.

So it would be nice if this enhancement is been made, it should be customizable when the flush occures.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/KonstantinCodes/messenger-kafka/issues/9#issuecomment-625203953, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZEDLQBDLYJKKWD326TJS3RQKNHHANCNFSM4LFFGVAA .

[tifabien]

@KonstantinCodes @rdotter You can have a look at my PR just created. It implements what you proposed in this ticket (flush on terminate or flush message per message).