SwayDB
SwayDB copied to clipboard
simerplaha
Never ending loop "Competing reserved resource accessed via runSync"
My use case is very easy, I'm using off-heap map to store some counters. Read and write is single threaded.
Very frequently my application is not functioning correctly because it cannot acces collection - the only sign that something is not working correctly is following log.
15:14:04.705 WARN swaydb.IO$Defer - io-compute-10: Competing reserved resource accessed via runSync. Times accessed: 3843425. Reserve: Some(ClosedChannel)
This counter goes to the moon 3843425
Is there any way to set some configuration parameters to avoid such situations - maybe by disabling compaction etc
Because my use case is very simple - the only compeeting process could be some swaydb -internal compaction etc.
Bellow you can find the config that I'm currently using.
/*
Specifies if key-value IDs should be cached. There are over 1300 key-value Ids, if caching is disabled then on disk binary-search is performed to find the IDs.
https://swaydb.io/configuration/cacheKeyValueIds/?language=scala
*/
val cacheKeyValueIdsOverride = false
/*
This enables caching raw bytes stored in Persistent Segment and disables caching of actual key-values
https://swaydb.io/configuration/memoryCache/?language=scala
*/
val memoryCacheOverride = MemoryCache.off
/*
Setting mmap to true in LevelZero will write key-values to memory-mapped write-ahead log files.
If false, java.nio.FileChannel are used.
https://swaydb.io/configuration/mmap/map/?language=scala
*/
val mmapDisabled = MMAP.Off(
ForceSave.Off
)
val threadStateCacheOverride = ThreadStateCache.Limit(hashMapMaxSize = 10, maxProbe = 1)
//ThreadStateCache.off
/*
Configures all Persistent Segment file for the Level
https://swaydb.io/configuration/segmentConfig/?language=scala
*/
val segmentConfigOverride = DefaultConfigs
.segmentConfig()
.copyWithMmap(
mmapDisabled
)
.copyWithCacheSegmentBlocksOnCreate(false)
.copyWithFileOpenIOStrategy(IOStrategy.AsyncIO(cacheOnAccess = true))
.copyWithBlockIOStrategy(
blockIOStrategy = (_) => IOStrategy.AsyncIO(cacheOnAccess = false)
)
// Following configuration disable caching on the file read/write layer
val fileCacheOverride = FileCache.On(
0,
ActorConfig.TimeLoop(
name = s"${this.getClass.getName} - FileCache TimeLoop Actor",
delay = 1.seconds,
ec = DefaultExecutionContext.sweeperEC
)
)
/*
It stores all keys in sorted order.
https://swaydb.io/configuration/sortedKeyIndex/?language=scala&q=sortedKeyIndex
*/
val sortedKeyIndexOverride = DefaultConfigs.sortedKeyIndex(false)
/*
hashIndexes can double random read performance and reduce IOps which would increases overall DB performance.
https://swaydb.io/configuration/randomKeyIndex/?language=scala
*/
val randomSearchIndexOverride = DefaultConfigs.randomSearchIndex(false)
/*
It allows for faster read performance over sortedKeyIndex or linear-search for random reads and is essential for fast forward & reverse iterations.
https://swaydb.io/configuration/binarySearchIndex/?language=scala
*/
val binarySearchIndexOverride = DefaultConfigs.binarySearchIndex(false)
/*
BloomFilter is a small byte array that can be created in each persistent segment and is used to determine if a key exists in the segment without actually searching the Segment.
https://swaydb.io/configuration/mightContainKeyIndex/?language=scal
*/
val mightContainIndexOverride = DefaultConfigs.mightContainIndex(false)
/*
Configures storage for values within a persistent segment.
https://swaydb.io/configuration/valuesConfig/?language=scala&q=mightContainIndex
*/
val valuesConfigOverride = DefaultConfigs.valuesConfig(false)
def mapCreate(name: String) =
persistent.Map[String, Int, Nothing, Glass](
dir = File.newTemporaryDirectory(name).deleteOnExit().path,
mmapMaps = mmapDisabled,
cacheKeyValueIds = cacheKeyValueIdsOverride,
segmentConfig = segmentConfigOverride,
sortedKeyIndex = sortedKeyIndexOverride,
randomSearchIndex = randomSearchIndexOverride,
binarySearchIndex = binarySearchIndexOverride,
mightContainIndex = mightContainIndexOverride,
valuesConfig = valuesConfigOverride,
fileCache = fileCacheOverride,
memoryCache = memoryCacheOverride,
)
Hey! This is due to the same issue you reported before (#358). In this case, a read (maybe compaction) is accessing an Async configured operation (IOStrategy.AsyncIO) which also is being concurrently accessed by another thread.
I think if you made all your configurations use IOStrategy.SynchronisedIO, this will not occur. In the settings you've shared, change IOStrategy.AsyncIO to IOStrategy.SynchronisedIO.
Hope this resolves the problem.
After giving this some more thought, I'm afraid it might not fully resolve the problem you are having.
This will be fully resolved in issue #318.
@simerplaha yeah it is not resolving the issue
It completly stagnate here
io-compute-2" #11 daemon prio=5 os_prio=0 cpu=1391856.78ms elapsed=6212.23s tid=0x00007f3860dd3800 nid=0x18 runnable [0x00007f38485a5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.pread0([email protected]/Native Method)
at sun.nio.ch.FileDispatcherImpl.pread([email protected]/FileDispatcherImpl.java:54)
at sun.nio.ch.IOUtil.readIntoNativeBuffer([email protected]/IOUtil.java:274)
at sun.nio.ch.IOUtil.read([email protected]/IOUtil.java:245)
at sun.nio.ch.FileChannelImpl.readInternal([email protected]/FileChannelImpl.java:811)
at sun.nio.ch.FileChannelImpl.read([email protected]/FileChannelImpl.java:796)
at swaydb.core.io.file.ChannelFile.read(ChannelFile.scala:94)
at swaydb.core.io.file.DBFile.read(DBFile.scala:412)
at swaydb.core.io.reader.FileReader.read(FileReader.scala:73)
at swaydb.core.segment.format.a.block.reader.BlockReader$.read(BlockReader.scala:86)
at swaydb.core.segment.format.a.block.reader.BlockReaderBase.read(BlockReaderBase.scala:62)
at swaydb.core.segment.format.a.block.reader.BlockReaderBase.read$(BlockReaderBase.scala:61)
at swaydb.core.segment.format.a.block.reader.UnblockedReader.read(UnblockedReader.scala:81)
at swaydb.core.segment.format.a.block.reader.BlockReaderBase.readRemaining(BlockReaderBase.scala:74)
at swaydb.core.segment.format.a.block.reader.BlockReaderBase.readRemaining$(BlockReaderBase.scala:73)
at swaydb.core.segment.format.a.block.reader.UnblockedReader.readRemaining(UnblockedReader.scala:81)
at swaydb.core.segment.format.a.block.segment.footer.SegmentFooterBlock$.read(SegmentFooterBlock.scala:192)
at swaydb.core.segment.format.a.block.segment.SegmentBlockCache.$anonfun$footerBlockCache$3(SegmentBlockCache.scala:348)
at swaydb.core.segment.format.a.block.segment.SegmentBlockCache.$anonfun$footerBlockCache$2(SegmentBlockCache.scala:347)
at swaydb.core.segment.format.a.block.segment.SegmentBlockCache$$Lambda$3318/0x0000000841347040.apply(Unknown Source)
at swaydb.data.cache.SynchronisedIO.$anonfun$value$7(Cache.scala:323)
at swaydb.data.cache.SynchronisedIO$$Lambda$1918/0x0000000840bd8840.apply(Unknown Source)
at swaydb.data.cache.LazyIO.$anonfun$getOrSet$3(Lazy.scala:165)
at swaydb.data.cache.LazyIO$$Lambda$1919/0x0000000840bd8c40.apply(Unknown Source)
at swaydb.data.cache.LazyValue.$anonfun$getOrSet$2(Lazy.scala:96)
at swaydb.data.cache.LazyValue$$Lambda$533/0x00000008403f7840.apply(Unknown Source)
at scala.Option.getOrElse(Option.scala:189)
at swaydb.data.cache.LazyValue.$anonfun$getOrSet$1(Lazy.scala:95)
- locked <0x00000000ea6e5ce8> (a swaydb.data.cache.LazyValue)
at swaydb.data.cache.LazyValue$$Lambda$532/0x00000008403f7440.apply(Unknown Source)
at scala.Option.getOrElse(Option.scala:189)
at swaydb.data.cache.LazyValue.getOrSet(Lazy.scala:93)
at swaydb.data.cache.LazyIO.getOrSet(Lazy.scala:165)
at swaydb.data.cache.SynchronisedIO.value(Cache.scala:323)
at swaydb.core.segment.format.a.block.segment.SegmentBlockCache.$anonfun$getFooter$1(SegmentBlockCache.scala:439)
at swaydb.core.segment.format.a.block.segment.SegmentBlockCache$$Lambda$3343/0x000000084135c840.apply(Unknown Source)
at scala.Option.getOrElse(Option.scala:189)
at swaydb.data.cache.LazyValue.getOrElse(Lazy.scala:126)
at swaydb.data.cache.LazyIO.getOrElse(Lazy.scala:175)
at swaydb.data.cache.SynchronisedIO.getOrElse(Cache.scala:326)
at swaydb.core.segment.format.a.block.segment.SegmentBlockCache.getFooter(SegmentBlockCache.scala:439)
at swaydb.core.segment.SegmentRef$.higher(SegmentRef.scala:330)
at swaydb.core.segment.PersistentSegmentOne.higher(PersistentSegmentOne.scala:362)
at swaydb.core.segment.PersistentSegmentOne.higher(PersistentSegmentOne.scala:219)
at swaydb.core.level.Level.higherFromFloorSegment(Level.scala:1324)
at swaydb.core.level.Level.higherInThisLevel(Level.scala:1344)
at swaydb.core.level.Level$$anon$1.higher(Level.scala:359)
at swaydb.core.level.seek.Higher$.apply(Higher.scala:99)
at swaydb.core.level.Level.higher(Level.scala:1368)
at swaydb.core.level.Level.swaydb$core$level$Level$$higherInNextLevel(Level.scala:1356)
at swaydb.core.level.Level$$anon$2.higher(Level.scala:372)
at swaydb.core.level.seek.Higher$.apply(Higher.scala:172)
at swaydb.core.level.Level.higher(Level.scala:1368)
at swaydb.core.level.Level.swaydb$core$level$Level$$higherInNextLevel(Level.scala:1356)
at swaydb.core.level.Level$$anon$2.higher(Level.scala:372)
at swaydb.core.level.seek.Higher$.apply(Higher.scala:172)
at swaydb.core.level.Level.higher(Level.scala:1368)
at swaydb.core.level.zero.LevelZero.findHigherInNextLevel(LevelZero.scala:792)
at swaydb.core.level.zero.LevelZero$$anon$4.higher(LevelZero.scala:837)
at swaydb.core.level.seek.Higher$.apply(Higher.scala:172)
at swaydb.core.level.seek.Higher$.seek(Higher.scala:69)
at swaydb.core.level.zero.LevelZero.findHigher(LevelZero.scala:864)
at swaydb.core.level.zero.LevelZero.higher(LevelZero.scala:881)
at swaydb.core.Core.$anonfun$after$1(Core.scala:309)
at swaydb.core.Core.after(Core.scala:176)
at swaydb.Map.swaydb$Map$$nextTupleOrNone(Map.scala:323)
at swaydb.Map$$anon$1.nextOrNull(Map.scala:337)
at swaydb.Map$$anon$1.nextOrNull(Map.scala:326)
at swaydb.data.stream.SourceFree.nextOrNull(SourceFree.scala:63)
at swaydb.data.stream.step.Map.nextOrNull(Map.scala:49)
at swaydb.data.stream.StreamFree.$anonfun$foldLeft$1(StreamFree.scala:226)
at swaydb.data.stream.StreamFree$$Lambda$3511/0x00000008413d8c40.apply(Unknown Source)
at swaydb.Bag$$anon$4.safe(Bag.scala:517)
at swaydb.data.stream.StreamFree.foldLeft(StreamFree.scala:226)
at swaydb.data.stream.StreamFree.foldLeft$(StreamFree.scala:218)
at swaydb.data.stream.step.Map.foldLeft(Map.scala:30)
at swaydb.data.stream.StreamFree.foreach(StreamFree.scala:244)
at swaydb.data.stream.StreamFree.foreach$(StreamFree.scala:243)
at swaydb.data.stream.step.Map.foreach(Map.scala:30)
at swaydb.Stream.foreach(Stream.scala:170)