numaflow
numaflow copied to clipboard
skip -1 watermark from source-transformer when calculating watermark at source
Summary
source-transformer might return -1 watermark for a message in a batch. since we also take min of watermarks, we will end up publishing -1 watermark which will be ignored. the downside of this is that the watermark will never progress if there is always a bad message in the batch.