numaflow icon indicating copy to clipboard operation
numaflow copied to clipboard

watermark updates gets linearly slow with increase in partition count

Open vigith opened this issue 7 months ago • 0 comments

Describe the bug watermark updates get linearly slow with an increase in partition count, causing forward-a-chunk to block. we noticed this issue when reading from Kafka with partition counts ~ 500.

Work Around Disabling the watermark can be done only for non-reduced pipelines.

vigith avatar Jul 27 '24 17:07 vigith