akashic-storage icon indicating copy to clipboard operation
akashic-storage copied to clipboard

memory fault kills the daemon

Open akiradeveloper opened this issue 8 years ago • 1 comments

I tried to put an object of size 160MB. the server's resource is tiny: memory 1GB and 1 cpu. Set smaller value for the put size makes sense but what size it should be?

To begin with, the memory allocation error shouldn't kill the daemon in any way. This showstopper. should be fixed before 1.0

Uncaught error from thread [akashic-storage-akka.actor.default-dispatcher-31] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[akashic-storage]
java.lang.OutOfMemoryError: Java heap space
        at scala.reflect.ManifestFactory$$anon$6.newArray(Manifest.scala:93)
        at scala.reflect.ManifestFactory$$anon$6.newArray(Manifest.scala:91)
        at scala.Array$.ofDim(Array.scala:218)
        at akka.util.ByteIterator.toArray(ByteIterator.scala:459)
        at akka.util.ByteString.toArray(ByteString.scala:430)
        at akka.http.scaladsl.unmarshalling.PredefinedFromEntityUnmarshallers$$anonfun$byteArrayUnmarshaller$1.apply(PredefinedFromEntityUnmarshallers.scala:20)
        at akka.http.scaladsl.unmarshalling.PredefinedFromEntityUnmarshallers$$anonfun$byteArrayUnmarshaller$1.apply(PredefinedFromEntityUnmarshallers.scala:20)
        at akka.http.scaladsl.util.FastFuture$$anonfun$map$extension$1.apply(FastFuture.scala:23)
        at akka.http.scaladsl.util.FastFuture$$anonfun$map$extension$1.apply(FastFuture.scala:23)
        at akka.http.scaladsl.util.FastFuture$.akka$http$scaladsl$util$FastFuture$$strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$$anonfun$transformWith$extension1$1.apply(FastFuture.scala:51)
        at akka.http.scaladsl.util.FastFuture$$anonfun$transformWith$extension1$1.apply(FastFuture.scala:50)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Service exit with a return value of 255

akiradeveloper avatar Mar 21 '16 04:03 akiradeveloper

I think this was because the daemon wasn't set up the max-content-length. it is by default MB so sending 160MB object will fault. But still strange it was killed by OOM

Anyway, this is not frequent in production. So will drop this off from v1

akiradeveloper avatar Mar 23 '16 12:03 akiradeveloper