Yashash H L

``` --- FAIL: TestFunctionalSuite (374.27s) --- FAIL: TestFunctionalSuite/TestConditionalForwarding (147.55s) --- PASS: TestFunctionalSuite/TestCreateSimplePipeline (64.10s) --- SKIP: TestFunctionalSuite/TestDropOnFull (0.00s) --- PASS: TestFunctionalSuite/TestUDFFiltering (82.37s) --- SKIP: TestFunctionalSuite/TestWatermarkEnabled (0.00s) ```

could be related to https://github.com/numaproj/numaflow/issues/1552

* Our system overly relies on the "Skip watermark publishing" log for data processing validation. If the watermark is disabled it's very difficult to validate. * The abundant debug logs...

Separate repository for proto files

2

# Problem definition Currently, we keep the proto files in all the SDKs, if we have to change anything we will need to manually update the proto files in all...

pipeline cleanup pod occasionally goes into crashloop state

1

``` {"level":"info","ts":"2023-07-21T08:06:56.613352112Z","logger":"numaflow.isbsvc-delete","caller":"isbsvc/jetstream_service.go:196","msg":"Succeeded to delete an offset timeline bucket","bucket":"dev-devx-osamonnumaflow-usw2-qal-osam-otlp-fixed-aggregation-sliding-aggregation_OT"} {"level":"info","ts":"2023-07-21T08:06:56.615230528Z","logger":"numaflow.isbsvc-delete","caller":"isbsvc/jetstream_service.go:201","msg":"Succeeded to delete a processor bucket","bucket":"dev-devx-osamonnumaflow-usw2-qal-osam-otlp-fixed-aggregation-sliding-aggregation_PROCESSORS"} {"level":"info","ts":"2023-07-21T08:07:01.616205689Z","logger":"numaflow.isbsvc-delete","caller":"nats/default_jetstream_client.go:63","msg":"Nats: disconnected"} {"level":"error","ts":"2023-07-21T08:07:01.61622148Z","logger":"numaflow.isbsvc-delete","caller":"commands/isbsvc_delete.go:66","msg":"Failed on buffers and buckets deletion.","error":"failed to delete offset timeline bucket \"dev-devx-osamonnumaflow-usw2-qal-osam-otlp-sliding-aggregation-sink_OT\", context deadline...

We're observing a high number of control messages in multi-partition pipelines, which is affecting efficiency.

``` {"level":"error","ts":"2024-07-27T15:13:08.068429208Z","logger":"numaflow.Sink-processor","caller":"fetch/processor_manager.go:269","msg":"Unable to find the processor","pipeline":"dummy-ceres-transaction-pipeline","vertex":"out","processorEntity":"dummy-ceres-transaction-pipeline-input-318","stacktrace":"github.com/numaproj/numaflow/pkg/watermark/fetch.(*processorManager).startTimeLineWatcher\n\t/home/runner/work/numaflow/numaflow/pkg/watermark/fetch/processor_manager.go:269"} {"level":"error","ts":"2024-07-27T15:13:08.068445148Z","logger":"numaflow.Sink-processor","caller":"fetch/processor_manager.go:269","msg":"Unable to find the processor","pipeline":"dummy-ceres-transaction-pipeline","vertex":"out","processorEntity":"dummy-ceres-transaction-pipeline-input-353","stacktrace":"github.com/numaproj/numaflow/pkg/watermark/fetch.(*processorManager).startTimeLineWatcher\n\t/home/runner/work/numaflow/numaflow/pkg/watermark/fetch/processor_manager.go:269"} panic: runtime error: index out of range [4] with length 1 goroutine 208 [running]: github.com/numaproj/numaflow/pkg/watermark/fetch.(*processorManager).startTimeLineWatcher(0xc0004cc300) /home/runner/work/numaflow/numaflow/pkg/watermark/fetch/processor_manager.go:296 +0xc70 created by...

Configuring Timeout for Graceful Shutdown

3

### Version latest ### Description I'm currently working with tonic for implementing gRPC services in Rust, and I've encountered a situation where the server might not shut down gracefully due...

Display the sink invocation [latency](https://github.com/numaproj/numaflow/blob/ac7b33b803c2530df6c7f24252fb4e714c9bd07f/rust/numaflow-core/src/monovertex/metrics.rs#L61) and end to end processing [latency](https://github.com/numaproj/numaflow/blob/ac7b33b803c2530df6c7f24252fb4e714c9bd07f/rust/numaflow-core/src/monovertex/metrics.rs#L57) metrics (P99, P95, P90, P50) for a monovertex in the Numaflow UI for default time period of 30 mins.