pranadb icon indicating copy to clipboard operation
pranadb copied to clipboard

Rate limiting for disk read/write etc

Open purplefox opened this issue 2 years ago • 0 comments

A Prana node is subject to certain limits - usually imposed by the hardware or devices. Examples are CPU processing, disk write throughput, disk read throughput, iops, network bandwidth etc.

For a node to run stably it's important that one task does not saturate any one of these as that can cause starvation for other tasks which need to access them. E.g. if we're ingesting rows too fast we could max out disk throughput - this can make it impossible for a new node to join the cluster as raft groups would need to catch up so would need to read old data and write it faster than the ingest rate to catch up.

We should provide limits for various fixed resources, and fix the limit at some level less than the maximum capacity, e.g. 75%. E.g. if max disk throughput is 100 MB/s then we should limit it to 75MB/s - this gives some spare throughput for other purposes.

We can look at either wrapping the Uber rate limiter that we already used for limiting source ingest (this would be replaced) or write our own one. It would be nice to be able to take multiple tokens in one call to avoid too much contention on the limiter, and also to be able to provide the minimum sleep time (for ingest, we are not latency sensitive)

purplefox avatar Mar 22 '22 08:03 purplefox