numaflow
numaflow copied to clipboard
Certain scenarios cause Buffer cleanup job to run at the same time as the Creation job - race condition
Describe the bug
~~While the ISB Batch Jobs for creating and cleaning ISB buckets and buffers are Owned by the Pipeline, the Pods themselves aren't.~~ (Striking through this part after clarification from @whynowy below in comment)
The result is that if you delete a Pipeline and then re-create the same Pipeline immediately after, the "clean" Job Pod from the first one could be running at the same time that the "create" Job Pod from the second one is running, which can cause the second one to have its buckets and buffers removed and then the Pipeline Pods to be stuck in Pending state waiting indefinitely for those buckets/buffers to be created.
Message from the maintainers:
Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.
For quick help and support, join our slack channel.