numaflow icon indicating copy to clipboard operation
numaflow copied to clipboard

Document "Dropping Messages" concept

Open th0ger opened this issue 9 months ago • 1 comments

In SDK implementations and event time filter examples there exists MessageToDrop(...) (go) and to_drop(...) (python).

It is not obvious why dropping messages must be done by adding dropped message with timestamps. (I have my guesses though.)

Please clarify in core numaflow docs the overall concept of:

  • why messages dropped in a particular way in stream processing
  • how it works/implemented in numaflow,
  • in which use cases it is relevant (other than sourcetransformer?)
  • if it's required or optional.

I'll be happy to review a doc PR to check if it makes sense.


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

th0ger avatar May 24 '24 11:05 th0ger