shark-cli neo4j is failing with the `org.neo4j.kernel.impl.util.collection.MemoryAllocationLimitException`

[lukmccall]

Description

I was testing the experimental neo4j command inside of the shark-cli tool on a fairly large application. Unfortunately, the process crashed with the org.neo4j.kernel.impl.util.collection.MemoryAllocationLimitException exception.

Steps to Reproduce

• Run shark-cli --device emulator-5554 --process <your_app> neo4j on a fairly memory-consuming application. I used an app built using React Native.
• You should see an org.neo4j.kernel.impl.util.collection.MemoryAllocationLimitException exception was thrown during the progress edges phase.

Expected behavior:

neo4j database should be created correctly.

Version Information

• shark-cli version: 2.10
• LeakCanary version: none
• Android OS version: 12

Additional Information

Stacktrace

Progress edges: 0%
Progress edges: 10%
Progress edges: 20%
Progress edges: 30%
Progress edges: 40%
Progress edges: 50%
Exception in thread "main" org.neo4j.kernel.impl.util.collection.MemoryAllocationLimitException: Can't allocate extra 512 bytes due to exceeding memory limit; used=2147483648, max=2147483648
	at org.neo4j.kernel.impl.util.collection.CapacityLimitingBlockAllocatorDecorator.allocate(CapacityLimitingBlockAllocatorDecorator.java:50)
	at org.neo4j.kernel.impl.util.collection.OffHeapMemoryAllocator.allocate(OffHeapMemoryAllocator.java:48)
	at org.neo4j.kernel.impl.util.collection.LinearProbeLongLongHashMap.allocateMemory(LinearProbeLongLongHashMap.java:775)
	at org.neo4j.kernel.impl.util.collection.LinearProbeLongLongHashMap.<init>(LinearProbeLongLongHashMap.java:107)
	at org.neo4j.kernel.impl.util.collection.OffHeapCollectionsFactory.newValuesMap(OffHeapCollectionsFactory.java:71)
	at org.neo4j.kernel.impl.api.state.EntityStateImpl.addProperty(EntityStateImpl.java:110)
	at org.neo4j.kernel.impl.api.state.TxState.relationshipDoReplaceProperty(TxState.java:474)
	at org.neo4j.kernel.impl.newapi.Operations.relationshipSetProperty(Operations.java:1185)
	at org.neo4j.cypher.internal.runtime.interpreted.TransactionBoundQueryContext$RelationshipWriteOperations.setProperty(TransactionBoundQueryContext.scala:424)
	at org.neo4j.cypher.internal.planning.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations.setProperty(ExceptionTranslatingQueryContext.scala:514)
	at org.neo4j.cypher.internal.runtime.interpreted.UpdateCountingQueryContext$CountingOps.setProperty(UpdateCountingQueryContext.scala:272)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.BaseCreatePipe.setProperty(CreatePipe.scala:76)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.BaseCreatePipe.$anonfun$setProperties$1(CreatePipe.scala:58)
	at org.neo4j.values.virtual.MapValue$MapWrappingMapValue.foreach(MapValue.java:127)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.BaseCreatePipe.setProperties(CreatePipe.scala:58)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.EntityCreatePipe.$anonfun$createRelationship$1(CreatePipe.scala:113)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.EntityCreatePipe.$anonfun$createRelationship$1$adapted(CreatePipe.scala:113)
	at scala.Option.foreach(Option.scala:407)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.EntityCreatePipe.createRelationship(CreatePipe.scala:113)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.CreatePipe.$anonfun$internalCreateResults$3(CreatePipe.scala:143)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.CreatePipe.$anonfun$internalCreateResults$3$adapted(CreatePipe.scala:142)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.CreatePipe.$anonfun$internalCreateResults$1(CreatePipe.scala:142)
	at org.neo4j.cypher.internal.runtime.ClosingIterator$$anon$3.next(ClosingIterator.scala:154)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.EmptyResultPipe.internalCreateResults(EmptyResultPipe.scala:30)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:82)
	at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:78)
	at org.neo4j.cypher.internal.runtime.interpreted.PipeExecutionResult.request(PipeExecutionResult.scala:69)
	at org.neo4j.cypher.internal.result.StandardInternalExecutionResult.request(StandardInternalExecutionResult.scala:90)
	at org.neo4j.cypher.internal.result.StandardInternalExecutionResult.initiate(StandardInternalExecutionResult.scala:53)
	at org.neo4j.cypher.internal.result.ClosingExecutionResult.$anonfun$initiate$1(ClosingExecutionResult.scala:58)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.neo4j.cypher.internal.result.ClosingExecutionResult.safely(ClosingExecutionResult.scala:99)
	at org.neo4j.cypher.internal.result.ClosingExecutionResult.initiate(ClosingExecutionResult.scala:58)
	at org.neo4j.cypher.internal.result.ClosingExecutionResult$.wrapAndInitiate(ClosingExecutionResult.scala:183)
	at org.neo4j.cypher.internal.CypherCurrentCompiler$CypherExecutableQuery.innerExecute(CypherCurrentCompiler.scala:442)
	at org.neo4j.cypher.internal.CypherCurrentCompiler$CypherExecutableQuery.execute(CypherCurrentCompiler.scala:379)
	at org.neo4j.cypher.internal.ExecutionEngine.doExecute(ExecutionEngine.scala:217)
	at org.neo4j.cypher.internal.ExecutionEngine.$anonfun$executeSubquery$1(ExecutionEngine.scala:178)
	at org.neo4j.cypher.internal.ExecutionEngine.closing(ExecutionEngine.scala:183)
	at org.neo4j.cypher.internal.ExecutionEngine.executeSubquery(ExecutionEngine.scala:175)
	at org.neo4j.cypher.internal.ExecutionEngine.execute(ExecutionEngine.scala:125)
	at org.neo4j.cypher.internal.javacompat.ExecutionEngine.executeQuery(ExecutionEngine.java:127)
	at org.neo4j.cypher.internal.javacompat.ExecutionEngine.executeQuery(ExecutionEngine.java:115)
	at org.neo4j.kernel.impl.coreapi.TransactionImpl.execute(TransactionImpl.java:285)
	at org.neo4j.kernel.impl.coreapi.TransactionImpl.execute(TransactionImpl.java:274)
	at shark.Neo4JCommand$Companion.dumpEdges(Neo4JCommand.kt:400)
	at shark.Neo4JCommand$Companion.dump(Neo4JCommand.kt:166)
	at shark.Neo4JCommand.run(Neo4JCommand.kt:97)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:139)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:14)
	at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:215)
	at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:212)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:230)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:253)
	at shark.MainKt.main(Main.kt:13)

I found two threads which may be helpful:

• https://community.neo4j.com/t5/neo4j-graph-platform/memory-limit-exceeded-while-running-embedded-database-server/m-p/17521
• https://community.neo4j.com/t5/neo4j-graph-platform/memory-issue-while-loading-relationships-can-t-allocate-512/m-p/30910

[lukmccall]

Also, here https://drive.google.com/file/d/1s_czM18IefTeZ84t7jgSwnaw-4g82s8s/view?usp=sharing is the process dump.

[pyricau]

Sounds like you need to bump the max heap?

[lukmccall]

I don't remember exactly, however, I think I've tried to use the -Xmx flag to increase the heap size. It didn't work in my case.

[episal]

Most likely your neo4j is configured :

• tx_state_memory_allocation --> to use "OFF_HEAP"

If that is the case, then by default the maximum amount of memory that can be allocated for all transactions at a given point in time, is set to be 2G.

Either you re-configure neo4j, to instead use "ON_HEAP", in this mode, neo4j will use "heap" memory to maintain tx state. Or you increase the current default for property:

• tx_state_max_off_heap_memory --> e.g. increase it to be 4G --> SettingValueParsers.BYTES.parse("4G")