numaflow
numaflow copied to clipboard
Document "Dropping Messages" concept
In SDK implementations and event time filter examples there exists MessageToDrop(...)
(go) and to_drop(...)
(python).
It is not obvious why dropping messages must be done by adding dropped message with timestamps. (I have my guesses though.)
Please clarify in core numaflow docs the overall concept of:
- why messages dropped in a particular way in stream processing
- how it works/implemented in numaflow,
- in which use cases it is relevant (other than sourcetransformer?)
- if it's required or optional.
I'll be happy to review a doc PR to check if it makes sense.
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.