Karim Radhouani

I was able to reproduce your issue, and I believe fix it in #235. The root cause is the amount of gNMI subscribe responses that get passed to the outputs...

Thanks for testing it. If you don't mind sharing: how many workers did you set? Did you test both `prometheus` and `prometheus_write`? How much was the memory usage bump ?

Added a few metrics to both outputs: ```text gnmic_prometheus_output_number_of_prometheus_metrics_total 200000 gnmic_prometheus_write_output_metadata_msg_send_duration_ns 949876 gnmic_prometheus_write_output_msg_send_duration_ns 909873 gnmic_prometheus_write_output_number_of_prometheus_write_metadata_msgs_sent_fail_total{reason=""} 0 gnmic_prometheus_write_output_number_of_prometheus_write_metadata_msgs_sent_success_total 200 gnmic_prometheus_write_output_number_of_prometheus_write_msgs_sent_fail_total{reason=""} 0 gnmic_prometheus_write_output_number_of_prometheus_write_msgs_sent_success_total 1e+06 ```

I think having more writes towards prometheus than incoming gRPC messages makes sense. A single Subscribe Response can hold multiple updates and those updates could have JSON values that map...

Yes workers are independent, they handle incoming messages by converting them to prometheus metrics and write them to a buffer. Then there is a single goroutine that drains the buffer...

> Can that single goroutine that drains the buffer be a bottleneck? Just setting a low interval e.g 1s or even sub-second i dont feel is enough to drain the...

> > > Can that single goroutine that drains the buffer be a bottleneck? Just setting a low interval e.g 1s or even sub-second i dont feel is enough to...

@Sparc0 if you build from #247 , you can configure the number of writers with `num-writers: 10` All the writers will be triggered periodically every `interval`. I could also add...

I believe that's the out of order delivery error, it can happen with multiple workers as well. Maybe I should create a buffer per series and reorder before writing to...

If it's a sliding window I think it should be enough to set it slightly higher than twice the prometheus_write output `interval` value.