numaflow icon indicating copy to clipboard operation
numaflow copied to clipboard

skip -1 watermark from source-transformer when calculating watermark at source

Open vigith opened this issue 1 year ago • 0 comments

Summary

source-transformer might return -1 watermark for a message in a batch. since we also take min of watermarks, we will end up publishing -1 watermark which will be ignored. the downside of this is that the watermark will never progress if there is always a bad message in the batch.

vigith avatar Dec 18 '23 16:12 vigith