Extremely slow queries on heavy write load

[danielschonfeld]

I seem to be running into a situation when doing heavy writes to the neo database and subsequently to the spatial "DB". I've put together some code I use to run a sample query utilizing gremlin while the database is being bombarded with writes and then thread dumped it.

I was pretty shocked to learn that my code was getting stuck at the spatial.getLayer() method and even more so to see that every time you initialize RTreeIndex it tries to update the metadata and write the total number of geometries. That results in a long wait for a lock which is not readily available since the nodes are being hit on the other end with heavy writes.

Here's the sample code that currently gets stuck with a random amount of time (sometimes longer, sometimes shorter... I'm assuming it basically is dependent on the time it takes it to acquire the write lock)

import org.neo4j.graphdb.index.*
import org.neo4j.graphdb.*
import org.neo4j.index.lucene.*
import org.apache.lucene.search.*
import org.neo4j.gis.spatial.*
import com.vividsolutions.jts.geom.Coordinate
import org.neo4j.gis.spatial.pipes.GeoPipeline
import org.neo4j.gis.spatial.indexprovider.GeoPipeFlowHits

neo4j = g.getRawGraph()
spatial = new SpatialDatabaseService(neo4j)
layer = spatial.getLayer('venues')

And here is the thread dump appropriate to my code running:

Thread 4433: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Thread.sleep(long, int) @bci=57, line=340 (Compiled frame)
 - org.neo4j.kernel.impl.util.concurrent.LockWaitStrategies$3.apply(long) @bci=43, line=69 (Compiled frame)
 - org.neo4j.kernel.ha.lock.forseti.ForsetiClient.acquireExclusive(org.neo4j.kernel.impl.locking.Locks$ResourceType, long[]) @bci=180, line=189 (Compiled frame)
 - org.neo4j.kernel.impl.api.LockingStatementOperations.nodeSetProperty(org.neo4j.kernel.impl.api.KernelStatement, long, org.neo4j.kernel.api.properties.DefinedProperty) @bci=35, line=280 (Compiled frame)
 - org.neo4j.kernel.impl.api.OperationsFacade.nodeSetProperty(long, org.neo4j.kernel.api.properties.DefinedProperty) @bci=17, line=545 (Compiled frame)
 - org.neo4j.kernel.impl.core.NodeProxy.setProperty(java.lang.String, java.lang.Object) @bci=46, line=254 (Compiled frame)
 - org.neo4j.collections.rtree.RTreeIndex.saveCount() @bci=67, line=435 (Interpreted frame)
 - org.neo4j.collections.rtree.RTreeIndex.initIndexMetadata() @bci=111, line=399 (Interpreted frame)
 - org.neo4j.collections.rtree.RTreeIndex.<init>(org.neo4j.graphdb.GraphDatabaseService, org.neo4j.graphdb.Node, org.neo4j.collections.rtree.EnvelopeDecoder, int) @bci=54, line=63 (Interpreted frame)
 - org.neo4j.gis.spatial.LayerRTreeIndex.<init>(org.neo4j.graphdb.GraphDatabaseService, org.neo4j.gis.spatial.Layer, int) @bci=15, line=49 (Interpreted frame)
 - org.neo4j.gis.spatial.LayerRTreeIndex.<init>(org.neo4j.graphdb.GraphDatabaseService, org.neo4j.gis.spatial.Layer) @bci=5, line=45 (Interpreted frame)
 - org.neo4j.gis.spatial.DefaultLayer.initialize(org.neo4j.gis.spatial.SpatialDatabaseService, java.lang.String, org.neo4j.graphdb.Node) @bci=158, line=281 (Interpreted frame)
 - org.neo4j.gis.spatial.DefaultLayer.makeLayerInstance(org.neo4j.gis.spatial.SpatialDatabaseService, java.lang.String, org.neo4j.graphdb.Node, java.lang.Class) @bci=22, line=251 (Interpreted frame)
 - org.neo4j.gis.spatial.DefaultLayer.makeLayerFromNode(org.neo4j.gis.spatial.SpatialDatabaseService, org.neo4j.graphdb.Node) @bci=64, line=219 (Interpreted frame)
 - org.neo4j.gis.spatial.SpatialDatabaseService.getLayer(java.lang.String) @bci=84, line=106 (Interpreted frame)
 - org.neo4j.gis.spatial.SpatialDatabaseService$getLayer.call(java.lang.Object, java.lang.Object) @bci=17 (Interpreted frame)
 - Script12.run() @bci=87, line=14 (Interpreted frame)
 - com.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(java.lang.Class, javax.script.ScriptContext) @bci=181, line=219 (Interpreted frame)

[aabutaleb]

@danielschonfeld I'm having a similar problem with Spatial. Could you find any solution to this?