numaflow
numaflow copied to clipboard
numaproj
Kubernetes-native platform to run massively parallel data/streaming jobs
# Summary Expose controller logs in the UI # Use Cases Users find it challenging to debug and see if there are any issues with Numaflow controller --- **Message from...
# Summary Show the source partition watermark in the UI so it is easy to know which partition is causing the lag # Use Cases While debugging --- **Message from...
# Summary What change needs making?  # Use Cases When would you use this? --- **Message from the maintainers**: If you wish to see this enhancement implemented please add...
# Summary Similar to [Argo Workflows](https://argo-workflows.readthedocs.io/en/latest/scaling/#instance-id), support multi controller deployment in one cluster/namespace. Potential solution: a.) Controller will have a `instance-id` configuration b.) Controllers having a `instance-id` will only take...
# Summary Today, users are not aware of what a Numaflow pipeline is or the different types of vertices, and how they are integrated. The best way to introduce Numaflow...
# Summary Log the payload size when the source errors out with an exceeded max payload size. It helps to adjust the ISB spec based on the values observed #...
**Describe the bug** A clear and concise description of what the bug is. - We are able to set the replicas above the max scale limit defined. Eg: Max defined...
Autoscaling behavior is not very effective out of the box for applications with slower startup times
# Summary The goal for our service is to be able to smoothly handle incoming event traffic from a Kafka Topic with high traffic (TPS >8k). One other thing to...
# Summary With current autoscaling [strategy](https://numaflow.numaproj.io/specs/autoscaling/), a high throughput pipeline (e.g. lots of backlog need to process), when the pipeline processing rate hits the bottleneck (due to ISB or anything...
# Summary Today, when there's any vertex pod recreation needed, we do a `Recreate` strategy, which means, we clean up all the old pods, and recreate everybody. This will cause...