JanusGraph
"Could not find type for id" error when traversing graph / Corrupted label in a vertex
The schema is pretty simple (few labels, properties, constraints and indexes).
- JanusGraph 0.4.1
- Storage Backend: Scylla 3.2.0
- 40k vertices
For me it looks like one day in the database appears a broken vertex which makes impossible to run any queries that iterate all vertices and contains "hasLabel()" method. We are still investigating this issue, but it does not seem like any schema changes were made right before we noticed it last time. This is how it looks:
gremlin> g.V().hasLabel('userProduct')
12:05:44 WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [(~label = userProduct)]. For better performance, use indexes
Could not find type for id: 155149
Type ':help' or ':h' for help.
gremlin> g.V().hasLabel('nnname')
12:49:34 WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [(~label = nnname)]. For better performance, use indexes
Could not find type for id: 155149
Type ':help' or ':h' for help.
Display stack trace? [yN]
Meanwhile method "has()" works fine:
gremlin>
gremlin> g.V().has('name', 'ddd')
12:24:37 WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [(name = ddd)]. For better performance, use indexes
==>v[368767136]
==>v[123867144]
gremlin>
When trying to get that vertex properties we get only ([id:155149,label:vertex]):
gremlin> g.V(155149)
Could not find type for id: 155149
Type ':help' or ':h' for help.
Display stack trace? [yN]
gremlin>
gremlin> g.V(155149).count()
==>1
gremlin> g.V(155149).elementMap()
==>[id:155149,label:vertex]
gremlin>
P.S. The problem is critical and already reached our production database...:((
It would be good if you add the stacktrace
java.lang.NullPointerException: Could not find type for id: 155149
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:250)
at org.janusgraph.graphdb.types.vertices.JanusGraphSchemaVertex.name(JanusGraphSchemaVertex.java:57)
at org.janusgraph.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:121)
at org.janusgraph.graphdb.types.system.ImplicitKey.computeProperty(ImplicitKey.java:83)
at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.executeImplicitKeyQuery(BasicVertexCentricQueryBuilder.java:211)
at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:99)
at org.janusgraph.graphdb.util.ElementHelper.getValues(ElementHelper.java:41)
at org.janusgraph.graphdb.query.condition.PredicateCondition.evaluate(PredicateCondition.java:68)
at org.janusgraph.graphdb.query.condition.And.evaluate(And.java:55)
at org.janusgraph.graphdb.query.graph.GraphCentricQuery.matches(GraphCentricQuery.java:153)
at org.janusgraph.graphdb.query.QueryProcessor.lambda$getFilterIterator$2(QueryProcessor.java:133)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:652)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54)
at org.janusgraph.graphdb.query.ResultSetIterator.<init>(ResultSetIterator.java:44)
at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66)
at com.google.common.collect.Iterables$7.iterator(Iterables.java:613)
at org.janusgraph.graphdb.tinkerpop.optimize.JanusGraphStep.executeGraphCentryQuery(JanusGraphStep.java:156)
at org.janusgraph.graphdb.tinkerpop.optimize.JanusGraphStep.lambda$null$1(JanusGraphStep.java:95)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.janusgraph.graphdb.tinkerpop.optimize.JanusGraphStep.lambda$new$2(JanusGraphStep.java:95)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep.processNextStart(GraphStep.java:157)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:197)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:234)
at org.apache.tinkerpop.gremlin.console.Console$_closure3.doCall(Console.groovy:255)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:37)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
at org.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:463)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:201)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:83)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:120)
at org.codehaus.groovy.tools.shell.Shell$leftShift$1.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:93)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:164)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:160)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:164)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:97)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:234)
at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:168)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:234)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:502)
@farodin91
One more interesting detail which could help. "ID" and "label" of the vertex are only returned if the error raised in current transaction:
gremlin> g.tx().rollback()
==>null
gremlin>
gremlin> g.V(155149).elementMap()
gremlin>
gremlin> g.V().hasLabel('nnname')
08:43:40 WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [(~label = nnname)]. For better performance, use indexes
Could not find type for id: 155149
Type ':help' or ':h' for help.
Display stack trace? [yN]
gremlin>
gremlin> g.V(155149).elementMap()
==>[id:155149,label:vertex]
gremlin>
gremlin>
gremlin> g.tx().rollback()
==>null
gremlin>
gremlin> g.V(155149).elementMap()
gremlin>
We have seen this as well, and wound up compiling our own version of JG (specifically, just the JanusGraphSchemaVertex class) that replaced that null pointer exception with code that returned "missing" (string literal) for the name instead of throwing a null pointer exception, then using a local gremlin connection to query / investigate / remove the offending nodes.
It's a bit frustrating that this check explicitly prevents you from writing any queries (even removes) that would actually fix the issue. Seems like this should be a WARN followed by a blank string return instead of an NPE that blocks all the queries that would be required to actually fix this issue.
@PatrickRice-KSC
Thank you for your reply. Could you share a patch for that specific JG version ?
@vladuk - for sure. We patched version 0.3.3 as that's what we're running, but the code is still present in 0.4.1 and 0.5.2 unchanged, and the update should work on both versions (though obviously test prior to applying anything). The fix involves the name method of the JanusGraphSchemaVertex.class file.
@Override
public String name() {
if (name == null) {
JanusGraphVertexProperty<String> p;
if (isLoaded()) {
StandardJanusGraphTx tx = tx();
p = (JanusGraphVertexProperty) Iterables.getOnlyElement(RelationConstructor.readRelation(this,
tx.getGraph().getSchemaCache().getSchemaRelations(longId(), BaseKey.SchemaName, Direction.OUT),
tx), null);
} else {
p = Iterables.getOnlyElement(query().type(BaseKey.SchemaName).properties(), null);
}
//Preconditions.checkState(p!=null,"Could not find type for id: %s", longId());
if (p != null) {
name = p.value();
}
}
//assert name != null;
if (name !=null ) {
return JanusGraphSchemaCategory.getName(name);
} else {
return "missing";
}
}
You can see we've essentially just removed the precondition for now, along with the null assertion and replaced them with values if they're null to ensure that the gremlin can return appropriately. Long-term there should also likely be a WARN message that's printed here as well so that there is a state you can watch for if this issue happens in a prod environment, along with a more unique name for the field than "missing" (as it's theoretically possible that a user has a key named "missing", though such a key still would work with this fix, just probably confuse downstream code).
You can then either build your own JanusGraph from scratch, or we just built the core jar (use mvn jar in the janusgraph-core folder) and swap the core jar in your gremlin server to update the patch. We were trying to fix a prod issue (as it sounds like you guys might be) so we kinda did a quick-n-dirty to get it resolved.
We faced this issue twice. First time we faced it with JG-0.4.0 + Cassandra-3.11.4 Recently same was observed wtih JG-0.6.1 + Cassandra-4.0 (support for Cassandra 4.0 is not official yet)
Our observation was that query was failing only if we specify edge label while traversing and it was happening for particular edge label and not for all. e.g. Query not working: g.V().has("_type","type1").in("affects").valueMap("_type") Query working without edge label: g.V().has("_type","type1").in().valueMap("_type")
We tried to print vertex with id printed in stacktrace but no properties were printed except id and label. This vertex doesn't seem to be created by our code and is some internally created vertex as we specify our custom id and label while creating vertex. gremlin> g.V(1383445).elementMap() ==>[id:1383445,label:vertex]
Workaround: restarting service automatically resolved this issue for us. Looks like some rarely occuring bug.
Below is stack trace for reference:
java.lang.NullPointerException: Could not find type for id: 1383445 at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:971) ~[guava-30.1-jre.jar:?] at org.janusgraph.graphdb.types.vertices.JanusGraphSchemaVertex.name(JanusGraphSchemaVertex.java:68) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQueryWithoutProfile(BasicVertexCentricQueryBuilder.java:533) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.constructQuery(BasicVertexCentricQueryBuilder.java:447) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.vertices(VertexCentricQueryBuilder.java:119) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphVertexStep.flatMap(JanusGraphVertexStep.java:142) ~[janusgraph-core-0.6.1.jar:?] at org.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:49) ~[gremlin-core-3.5.1.jar:3.5.1] at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:150) ~[gremlin-core-3.5.1.jar:3.5.1] at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:55) ~[gremlin-core-3.5.1.jar:3.5.1] at org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:37) ~[gremlin-core-3.5.1.jar:3.5.1] at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:150) ~[gremlin-core-3.5.1.jar:3.5.1] at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:55) ~[gremlin-core-3.5.1.jar:3.5.1] at org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:37) ~[gremlin-core-3.5.1.jar:3.5.1] at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:150) ~[gremlin-core-3.5.1.jar:3.5.1] at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:222) ~[gremlin-core-3.5.1.jar:3.5.1]
Any more intenal details on RCA for this issue would help.
Thanks
I also encountered the same problem in version 0.6.2. After using 'bin/janusgraph.sh' to stop\start the service, the problem is still the same. Is there a way to fix it?
gremlin> g.V().hasLabel('userProduct') 12:05:44 WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [(~label = userProduct)]. For better performance, use indexes Could not find type for id: 155149 Type ':help' or ':h' for help.
gremlin> g.V().hasLabel('nnname') 12:49:34 WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [(~label = nnname)]. For better performance, use indexes Could not find type for id: 155149 Type ':help' or ':h' for help. Display stack trace? [yN]
it happens if multiple instance of gremlin-server are running it is because gremlin server was not shutdown or killed properly it can be because the vm on which gremlin-server is running might have restarted
so the solution is login to gremlin-console and run your commands based on your backend in my case it is cassandra and elasticsearch so i will run method 1 :remote connect tinkerpop.server conf/remote.yaml session :remote console session or graph=JanusGraphFactory.open('conf/janusgraph-cql-es.properties'); g=graph.traversal()
and if you are running containers then your command must be similar to this graph=JanusGraphFactory.open('/etc/opt/janusgraph/janusgraph.properties'); g=graph.traversal()
now after running those you can run mgmt = graph.openManagement() mgmt.getOpenInstances()
it will display all the instances eg ac12000231-a9ffbcbb0e921 ac12000230-a9ffbcbb0e921(current)
except that current instance close other instances mgmt.forceCloseInstance('ac12000231-a9ffbcbb0e921')
after closing all the instances commit the changes mgmt.commit()
now restart your gremlin server and run your query it should work
method 2 if the problem persists just kill your gremlin-server and start it again few times...it should work
another reason why this happens is if the data is not restored properly.. if you are using cluster take the backup on all the nodes then restore on your destination node or nodes
i used nodetool for backup and sstableloader for restoring data
