numaflow icon indicating copy to clipboard operation
numaflow copied to clipboard

Data Loss Scenario: Consumer metadata causes Limits policy to drop messages on the floor

Open vigith opened this issue 6 months ago • 0 comments

Describe the bug

Unprocessed == 0 while Stream seq in the stream metadata is moving ahead of the consumer seq number.

Consumer Metadata

Last Delivered Message: Consumer sequence: 221,160,045 Stream sequence: 264,153,054
    Acknowledgment Floor: Consumer sequence: 221,160,045 Stream sequence: 264,153,054
        Outstanding Acks: 0 out of maximum 25,000
    Redelivered Messages: 0
    Unprocessed Messages: 0

Stream Metadata

Messages: 100,000
                 Bytes: 670 MiB
        First Sequence: 226,498,356 @ 2024-07-31 18:05:40 UTC
         Last Sequence: 226,598,355 @ 2024-07-31 18:06:02 UTC

To Reproduce

It is yet to be figured out. This problem is seen only when we have OOMs on JetStream. The pipeline where we saw this problem was inserting 64MB messages at high TPS, causing OOMs.

### Tasks
- [ ] https://github.com/numaproj/numaflow/issues/1885
- [ ] https://github.com/numaproj/numaflow/issues/1886
- [ ] https://github.com/numaproj/numaflow/issues/1887
- [ ] https://github.com/numaproj/numaflow/issues/1889
- [ ] pipeline buffer length cannot be more than ISB maxMsg
- [ ] https://github.com/numaproj/numaflow/issues/1891
- [ ] https://github.com/numaproj/numaflow/issues/1893
- [ ] https://github.com/numaproj/numaflow/issues/1899
- [ ] https://github.com/numaproj/numaflow/issues/1900
- [ ] https://github.com/numaproj/numaflow/issues/1907
- [ ] https://github.com/numaproj/numaflow/issues/1551
- [ ] https://github.com/numaproj/numaflow/issues/1554

vigith avatar Jul 31 '24 18:07 vigith