numaflow
numaflow copied to clipboard
MonoVertex to bypass ISB for high TPS large payload use cases
Summary
Today, we have to have at least a Source and a Sink. This means we need to write the message from the Source vertex to the ISB so that the Sink vertex can pick it up. For simple processing needs, one could avoid having the ISB altogether and implement Source and Sink in the same Vertex. SourcerSinker cannot support exactly-once semantics if the source does not support it because the Ack will be done after the message has been written on Sink, and we can hit the 2-generals problem.
Use Cases
If you are okay with the at-least-once semantics and want to avoid the cost of writing to ISB when the processing function is cheap to execute.
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
### Tasks
- [ ] https://github.com/numaproj/numaflow/pull/1898
- [ ] https://github.com/numaproj/numaflow/issues/1903
- [x] exiting is not clean on ctrl-c
- [ ] https://github.com/numaproj/numaflow/issues/1901
- [x] rename sourcer-sinker to mono-vertex
- [ ] #1927
- [ ] https://github.com/numaproj/numaflow/issues/1925
- [ ] https://github.com/numaproj/numaflow/issues/1932
- [ ] https://github.com/numaproj/numaflow-rs/issues/77
- [ ] https://github.com/numaproj/numaflow/issues/1926
- [ ] https://github.com/numaproj/numaflow/issues/1923
- [ ] https://github.com/numaproj/numaflow/issues/1924
- [ ] Check if all build flags will be populated at runtime for MonoVertex version
- [ ] https://github.com/numaproj/numaflow/issues/2033
- [ ] copy proto files at build time
- [ ] #1945
- [ ] https://github.com/numaproj/numaflow/issues/1929
- [ ] https://github.com/numaproj/numaflow/issues/1936
- [ ] https://github.com/numaproj/numaflow/issues/1937
- [ ] https://github.com/numaproj/numaflow/issues/1940
- [ ] https://github.com/numaproj/numaflow/issues/1943
- [ ] https://github.com/numaproj/numaplane/issues/193
- [ ] https://github.com/numaproj/numaflow/issues/1950
- [ ] https://github.com/numaproj/numaflow/issues/1952
- [ ] https://github.com/numaproj/numaflow/issues/1960
- [ ] https://github.com/numaproj/numaflow/issues/1959
- [ ] https://github.com/numaproj/numaflow/issues/2090
- [ ] -ve test cases
- [ ] https://github.com/numaproj/numaflow/issues/1986
- [ ] https://github.com/numaproj/numaflow/issues/1988
- [ ] log better latency and processing metrics with histograms
- [ ] https://github.com/numaproj/numaflow/pull/1994
- [ ] https://github.com/numaproj/numaflow/issues/2005
- [x] metrics for latency per container (source, transformer, sink)
- [ ] https://github.com/numaproj/numaflow/issues/2038
- [ ] https://github.com/numaproj/numaflow/issues/2098