Need a way to set SubscriberIntervalInSeconds in millseconds and a config parameter for fetch.max.wait.ms

[BoutemineOualid]

Hi,

We have very strict SLA requirements with respect to the throughput & latency of our system and it seems like in some edge cases when the load is very low on the system, the function does not get triggered frequently enough to meet the minimum end-to-end transaction times. This is mainly attributed to the fact that we are using a large batch size to handle our normal use case but this becomes a problem when the number of processed messages is very low and do not meet the max batch size.

We tried forcing lower values of the SubscriberIntervalInSeconds but the problem is that we can't go below 1 second and specify floating point values for milliseconds even though the code uses TimeSpan.FromSeconds which accepts doubles. So, I am asking if it's possible to make this parameter accept double values or make it available in a millisec configuration. The only possible solution for our problem was to lower the max batch size to a very small value to force a flush in the poll loop which would improve the latency in such a scenario but would severely affect the throughput and thus the end to end latency when the load spikes suddenly (The load balancer will take a lot of time to spin up additional instances to meet this demand, hence the need for large batch size config).

The other problem we noticed on the exported Kafka metrics is that when the load is very low, the RemoteTimeMs metric is always 500ms which maps to the default value of the fetch.max.wait.ms parameter (FetchWaitMaxMs config). We are wondering if it's possible to expose it with the configs and let us control it along with the fetch.min.bytes parameter.

Thanks