Sidhant Kohli

On a recent exploration with `Limits` policy, I was seeing the memory spiking up considerably again. ``` │ Stream Report │ ├──────────────────────────────────────────────────────────────────────────────┬─────────┬───────────┬───────────┬──────────┬─────────┬──────┬─────────┬──────────────────────────────────────────────────────────────────┤ │ Stream │ Storage │ Placement │ Consumers...

@neilalexander We use the Jetstream publish ``` // PublishMsg publishes a Msg to JetStream. PublishMsg(m *Msg, opts ...PubOpt) (*PubAck, error) ``` https://github.com/nats-io/nats.go/blob/main/js.go#L45

I see a buildup during the writeMsgRecord as well [mem_prof_2.zip](https://github.com/user-attachments/files/17018141/mem_prof_2.zip) @neilalexander @derekcollison

@derekcollison @neilalexander Any pointer for the above?

@neilalexander Even though the max payload size is set at 64Mib The average payload size is 60kb ``` │ xxxx-simple-pipeline-out-1 │ File │ │ 1 │ 100,000 │ 6.2 GiB...

Each stream has 117 Mb of data ``` Information for Stream STREAM1 created 2024-08-01 14:50:04 Subjects: stream1.> Replicas: 3 Storage: File Options: Retention: WorkQueue Acknowledgments: true Discard Policy: New Duplicate...

For anyone looking at this, as a workaround please allocate same resources to init container and container template. That should not block the pod startup.

Hello @gopaljayanthi! From the logs I can see the error `2024/06/21 22:05:45 starting the gRPC server with unix domain socket... /var/run/numaflow/function.sock` This is an older version of the UDF which...

Thanks @sapkota-aayush for this up! Like always please feel free to discuss and we can take it forward :)

Hey @sapkota-aayush Let's start with sync server flow first! And ensure consistency between both Then we can take it further