dvid icon indicating copy to clipboard operation
dvid copied to clipboard

Limit concurrent access of single file system leveldb backends

Open DocSavage opened this issue 8 years ago • 0 comments

At high concurrency and large data access, single file system leveldb backends can be thrashed. For the storage/leveldb drivers, it would make sense to serialize access requests via an I/O queue and also return errors once a sufficient backlog occurs. This would be a fallback warning in case clients don't use the currently optional throttling mechanism.

Precipitating event: user made over 150 concurrent requests, unthrottled, for ~500x500x500 subvolumes within one minute. This caused extreme load and unresponsiveness in a Linux system with RAID-6 drive using basho-tuned leveldb.

DocSavage avatar Feb 10 '17 15:02 DocSavage