numaflow
numaflow copied to clipboard
Data Loss Scenario: Consumer metadata causes Limits policy to drop messages on the floor
Describe the bug
Unprocessed == 0 while Stream seq in the stream metadata is moving ahead of the consumer seq number.
Consumer Metadata
Last Delivered Message: Consumer sequence: 221,160,045 Stream sequence: 264,153,054
Acknowledgment Floor: Consumer sequence: 221,160,045 Stream sequence: 264,153,054
Outstanding Acks: 0 out of maximum 25,000
Redelivered Messages: 0
Unprocessed Messages: 0
Stream Metadata
Messages: 100,000
Bytes: 670 MiB
First Sequence: 226,498,356 @ 2024-07-31 18:05:40 UTC
Last Sequence: 226,598,355 @ 2024-07-31 18:06:02 UTC
To Reproduce
It is yet to be figured out. This problem is seen only when we have OOMs on JetStream. The pipeline where we saw this problem was inserting 64MB messages at high TPS, causing OOMs.
### Tasks
- [ ] https://github.com/numaproj/numaflow/issues/1885
- [ ] https://github.com/numaproj/numaflow/issues/1886
- [ ] https://github.com/numaproj/numaflow/issues/1887
- [ ] https://github.com/numaproj/numaflow/issues/1889
- [ ] pipeline buffer length cannot be more than ISB maxMsg
- [ ] https://github.com/numaproj/numaflow/issues/1891
- [ ] https://github.com/numaproj/numaflow/issues/1893
- [ ] https://github.com/numaproj/numaflow/issues/1899
- [ ] https://github.com/numaproj/numaflow/issues/1900
- [ ] https://github.com/numaproj/numaflow/issues/1907
- [ ] https://github.com/numaproj/numaflow/issues/1551
- [ ] https://github.com/numaproj/numaflow/issues/1554