dvid
dvid copied to clipboard
Limit concurrent access of single file system leveldb backends
At high concurrency and large data access, single file system leveldb backends can be thrashed. For the storage/leveldb drivers, it would make sense to serialize access requests via an I/O queue and also return errors once a sufficient backlog occurs. This would be a fallback warning in case clients don't use the currently optional throttling mechanism.
Precipitating event: user made over 150 concurrent requests, unthrottled, for ~500x500x500 subvolumes within one minute. This caused extreme load and unresponsiveness in a Linux system with RAID-6 drive using basho-tuned leveldb.