JanusGraph
Ineffective storage strategy in ConsistentKeyIDAuthority
- Version: 0.6.0
- Storage Backend: All
- Mixed Index Backend: none
Current implementation of ConsistentKeyIDAuthority stores all allocated blocks of ids. This lead to constant increase of data stored and eventually causes error in AerospikeStoreManager (Aerospike allows to store only up to 1Mb per record)
To allocate new block of ids ConsistentKeyIDAuthority reads all this history data from storage but uses only the last id entry
My proposal is to keep only the last allocated rangeblock and remove elder ones
I've checked the code and think it should improve performance (no need to read and compare all range blocks) and shouldn't add any side effects
I suspect this might create a problem in a highly concurrent environment.
https://github.com/JanusGraph/janusgraph/blob/1dd189d368f3429aaa3c6dc0a504dd608a6f4dca/janusgraph-core/src/main/java/org/janusgraph/diskstorage/idmanagement/ConsistentKeyIDAuthority.java#L299-L301
In the above code, we read all id allocations and see if we are the most senior claim. If not, then it means another claimant beats us and we need to try again. If the elder blocks were indeed claimed by others but have already been purged, then we might falsely assume we win the application while we don't.
Hi @li-boxuan. Good catch. Added logic to remove only old enough blocks that shouldn't take part in concurrent claim.