parallel-consumer icon indicating copy to clipboard operation
parallel-consumer copied to clipboard

With KEY ordering, option to combine queues from different partitions or topics

Open astubbs opened this issue 3 years ago • 3 comments

In key mode, if records of a given key exist in

  • multiple partitions of a topic (and those partitions are assigned to me)
  • or when subscribing to multiple topics which have matching keys, have an option to have these either combined into a single queue, or kept separate.

If combined, option to choose how records are ordered relatively - e.g. via time stamp extractor, offset or publish time stamp.

Related:

  • #184

astubbs avatar May 24 '22 16:05 astubbs

We are looking at similar use case where we have multiple topics (for prioritization and fairness usecases) and we wanted key grouping across all topics related to a microservice (wildcard topic or list of topics specified). I suppose the current key grouping is supported to work for messages for a given key at a single partition level - is this assumption correct?

I see the PR is closed as stale but did not have much comments if had any other issues. Any updates on the feasibility of this feature request?

phanivankadari avatar Aug 08 '23 15:08 phanivankadari