Requested array size exceeds VM limit when reloading data

[mattdornfeld]

I'm doing the following:

1. Creating an instance of SwayDB that saves about 4 GBs of data to the file system
2. Exiting the process then starting a new one, which attempts to recover the data

When I attempt step 2 with the following command

      persistent.MultiMap[String, Key, Value, Nothing, Glass](
        dir = path,
        mapSize = 1000.mb,

I get the below error. Any idea on the cause?

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
  at swaydb.core.io.file.ChannelFile.readAll(ChannelFile.scala:102)
  at swaydb.core.io.file.DBFile.readAll(DBFile.scala:435)
  at swaydb.core.map.PersistentMap$.$anonfun$recover$1(PersistentMap.scala:161)
  at swaydb.core.map.PersistentMap$$$Lambda$4724/0x0000000841903840.apply(Unknown Source)
  at swaydb.IO$IterableIOImplicit.mapRecover(IO.scala:254)
  at swaydb.core.map.PersistentMap$.recover(PersistentMap.scala:157)
  at swaydb.core.map.PersistentMap$.apply(PersistentMap.scala:64)
  at swaydb.core.map.Maps$.$anonfun$recover$3(Maps.scala:214)
  at swaydb.core.map.Maps$.doRecovery$1(Maps.scala:213)
  at swaydb.core.map.Maps$.recover(Maps.scala:261)
  at swaydb.core.map.Maps$.persistent(Maps.scala:89)
  at swaydb.core.level.zero.LevelZero$.$anonfun$apply$4(LevelZero.scala:129)
  at swaydb.core.level.zero.LevelZero$$$Lambda$4770/0x0000000841949840.apply(Unknown Source)
  at swaydb.IO$Right.flatMap(IO.scala:570)
  at swaydb.core.level.zero.LevelZero$.$anonfun$apply$1(LevelZero.scala:122)
  at swaydb.core.level.zero.LevelZero$$$Lambda$4768/0x0000000841939040.apply(Unknown Source)
  at swaydb.IO$Right.flatMap(IO.scala:570)
  at swaydb.core.level.zero.LevelZero$.apply(LevelZero.scala:104)
  at swaydb.core.CoreInitializer$.$anonfun$apply$6(CoreInitializer.scala:252)
  at swaydb.core.CoreInitializer$$$Lambda$4767/0x000000084193b040.apply(Unknown Source)
  at swaydb.IO$Right.flatMap(IO.scala:570)
  at swaydb.core.CoreInitializer$.createLevels$1(CoreInitializer.scala:245)
  at swaydb.core.CoreInitializer$.$anonfun$apply$12(CoreInitializer.scala:322)
  at swaydb.core.CoreInitializer$$$Lambda$4764/0x000000084193e040.apply(Unknown Source)
  at swaydb.IO$Right.flatMap(IO.scala:570)
  at swaydb.core.CoreInitializer$.createLevels$1(CoreInitializer.scala:320)
  at swaydb.core.CoreInitializer$.$anonfun$apply$12(CoreInitializer.scala:322)
  at swaydb.core.CoreInitializer$$$Lambda$4764/0x000000084193e040.apply(Unknown Source)
  at swaydb.IO$Right.flatMap(IO.scala:570)
  at swaydb.core.CoreInitializer$.createLevels$1(CoreInitializer.scala:320)
  at swaydb.core.CoreInitializer$.$anonfun$apply$12(CoreInitializer.scala:322)
  at swaydb.core.CoreInitializer$$$Lambda$4764/0x000000084193e040.apply(Unknown Source)

[simerplaha]

Was your configured mapSize greater than Int.MaxValue - 2 at any time? Int.MaxValue is too large. You will see the same error with new Array[Byte](Int.MaxValue).

#280 will resolve this but for now please use smaller file sizes. I would recommend <= 10.mb or just use the defaults.

We will have much better performance handling large files in the next release.

[simerplaha]

We have to limit file sizes for now. Caching and other processes make uses of byte arrays (Array[Byte]) which are limited to 2.14.GB on the JVM so file size limits are required.

I'm setting this limit to 1.GB for now. If required we can push this to 2.14.GB in future releases.