MapDB seems to cause invalid state in eclipse-collections LongObjectHashMap, Infinite loop in probeThree

[koenr-bc]

See also https://github.com/eclipse/eclipse-collections/issues/1134

Version info;

• mapdb: 3.0.8
• eclipse-collections: 10.4.0

In some cases the mapdb lib seems to be able to create an inconsistent state in the eclipse-collections LongObjectHashMap which causes an infinite loop. stack trace of the case;

    at org.eclipse.collections.impl.map.mutable.primitive.LongObjectHashMap.probeThree(LongObjectHashMap.java:3104)
    at org.eclipse.collections.impl.map.mutable.primitive.LongObjectHashMap.probeTwo(LongObjectHashMap.java:3080)
    at org.eclipse.collections.impl.map.mutable.primitive.LongObjectHashMap.probe(LongObjectHashMap.java:3057)
    at org.eclipse.collections.impl.map.mutable.primitive.LongObjectHashMap.getIfAbsent(LongObjectHashMap.java:2362)
    at org.eclipse.collections.impl.map.mutable.primitive.LongObjectHashMap.get(LongObjectHashMap.java:2340)
    at org.mapdb.StoreWAL.longStackLoadChunk(StoreWAL.kt:741)
    at org.mapdb.StoreWAL.longStackPut(StoreWAL.kt:723)
    at org.mapdb.StoreDirectAbstract.releaseData(StoreDirectAbstract.kt:367)
    at org.mapdb.StoreWAL.linkedRecordDelete(StoreWAL.kt:330)
    at org.mapdb.StoreWAL.updateProtected(StoreWAL.kt:449)
    at org.mapdb.StoreWAL.update(StoreWAL.kt:426)
    at org.mapdb.HTreeMap.putprotected(HTreeMap.kt:415)
    at org.mapdb.HTreeMap.put(HTreeMap.kt:324)

Case is using transactionEnable.

[martenk-bc]

Hi, I'm a colleague of Koen. We've encountered this issue a few times now. It seems to be preceded by the following error:

java.lang.ArrayIndexOutOfBoundsException: 1
	at org.eclipse.collections.impl.map.mutable.primitive.LongObjectHashMap.getIfAbsent(LongObjectHashMap.java:2365)
	at org.eclipse.collections.impl.map.mutable.primitive.LongObjectHashMap.get(LongObjectHashMap.java:2340)
	at org.mapdb.StoreWAL.longStackLoadChunk(StoreWAL.kt:741)
	at org.mapdb.StoreWAL.longStackTake(StoreWAL.kt:842)
	at org.mapdb.StoreDirectAbstract.allocateRecid(StoreDirectAbstract.kt:255)
	at org.mapdb.StoreWAL.put(StoreWAL.kt:383)
	at org.mapdb.HTreeMap.valueWrap(HTreeMap.kt:1208)
	at org.mapdb.HTreeMap.putprotected(HTreeMap.kt:344)
	at org.mapdb.HTreeMap.put(HTreeMap.kt:324)

I've tracked it down to a synchronization issue on StoreWAL.cachedStacks:

• When calling HTreeMap.put(), it locks locks[segment].writeLock in HTreeMap.put(), and structuralLock in StoreWAL.put() to call allocateRecid(). It then performs a cachedStacks.get() and cachedStacks.put().
• When calling DB.commit() or DB.rollback(), it acquires lock.writeLock() in DB.commit(), and all the write locks in locks in StoreWAL.commit(). This calls cachedStacks.clear() and cachedStacks.compact() in both cases, and cachedStacks.forEveryKeyValue() for a commit.

So here there are at least two distinct codepaths that both read and modify StoreWAL.cachedStacks, with a non-overlapping set of locks.