nats-server icon indicating copy to clipboard operation
nats-server copied to clipboard

Jetstream - Ack batch of messages while working with pull consumers

Open idanasulin2706 opened this issue 2 years ago • 2 comments

Feature Request

Currently when working with pull consumers we can only ack each message, as in Kafka I want to be able to ack the entire batch in order to minimize ack requests overhead

Use Case:

when working with large amount of messages ack should be call for each message

Proposed Change:

ackBatch instead of just ack

Who Benefits From The Change(s)?

I think that everyone

Proposing myself for contribute this code

idanasulin2706 avatar Jul 04 '22 11:07 idanasulin2706

Most clients do m.Ack() async, and the client will most likely batch them. Have you measured this and detected a performance problem?

We plan on allowing consumers to only have one active subscriber/puller at a time as an option. Once that is the case this could be opened up, for right now this could lead to some confusion as multiple subscribers will ack each other's outstanding messages.

derekcollison avatar Jul 04 '22 18:07 derekcollison

@idanasulinmemphis In the general case (more than one client instance subscribing to a JS Consumer), the consumer delivery sequence numbers will not be consecutive at any one instance as @derekcollison alludes. We don't currently track un-acknowledged deliveries by request/batch.

Although not yet integrated into client libraries, there is ability in the server to replace separate ACK message and a new request/batch message into a single message (this is called "AckNext"). This is probably the next step in reduction of over the wire signaling messages between client and server (your interest if I follow).

tbeets avatar Jul 05 '22 04:07 tbeets

For consumers, there is the option to set the AckAll policy as part of the configuration. This is indeed consumer specific, rather than an ad-hoc msg ack option, but should satisfy the use case for the particular workload.

bruth avatar Jun 11 '23 19:06 bruth