SchemaViolationException error when importing CPG

[GiuliCler]

Hi all, I obtain this error by importing (importCpg command) a CPG in joern environment. I have already increased the heap size, and as additional information the CPG is around 718M of size.

...
2022-05-05 04:33:56.874 WARN MethodRefLinker: Invalid AST_PARENT_TYPE=Optional[]; astChild LABEL=TYPE_DECL; astChild FULL_NAME=Optional[<function_name>]
2022-05-05 04:33:56.874 WARN MethodRefLinker: Invalid AST_PARENT_TYPE=Optional[]; astChild LABEL=TYPE_DECL; astChild FULL_NAME=Optional[<function_name>]
2022-05-05 04:33:56.874 WARN MethodRefLinker: Invalid AST_PARENT_TYPE=Optional[]; astChild LABEL=TYPE_DECL; astChild FULL_NAME=Optional[<function_name>]
overflowdb.SchemaViolationException: OUT edge with label AST to an adjacent METHOD_RETURN is mandatory, but not defined for this METHOD node with id=2305843009224193225
  io.shiftleft.codepropertygraph.generated.nodes.MethodDb.methodReturn(Method.scala:669)
  io.shiftleft.codepropertygraph.generated.nodes.Method.methodReturn(Method.scala:320)
  io.joern.x2cpg.passes.controlflow.cfgcreation.CfgCreator.<init>(CfgCreator.scala:55)
  io.joern.x2cpg.passes.controlflow.CfgCreationPass.runOnPart(CfgCreationPass.scala:23)
  io.joern.x2cpg.passes.controlflow.CfgCreationPass.runOnPart(CfgCreationPass.scala:17)
  io.shiftleft.passes.ConcurrentWriterCpgPass.$anonfun$createApplySerializeAndStore$2(ParallelCpgPass.scala:248)
  scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
  scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
  java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
  java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
  java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
  java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
  java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
  java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
java.util.NoSuchElementException: next on empty iterator
  scala.collection.Iterator$$anon$19.next(Iterator.scala:966)
  scala.collection.Iterator$$anon$19.next(Iterator.scala:964)
  scala.collection.Iterator$$anon$7.next(Iterator.scala:528)
  overflowdb.traversal.Traversal.next(Traversal.scala:22)
  io.shiftleft.codepropertygraph.generated.nodes.MethodDb.methodReturn(Method.scala:664)
  io.shiftleft.codepropertygraph.generated.nodes.Method.methodReturn(Method.scala:320)
  io.joern.x2cpg.passes.controlflow.cfgcreation.CfgCreator.<init>(CfgCreator.scala:55)
  io.joern.x2cpg.passes.controlflow.CfgCreationPass.runOnPart(CfgCreationPass.scala:23)
  io.joern.x2cpg.passes.controlflow.CfgCreationPass.runOnPart(CfgCreationPass.scala:17)
  io.shiftleft.passes.ConcurrentWriterCpgPass.$anonfun$createApplySerializeAndStore$2(ParallelCpgPass.scala:248)
  scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
  scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
  java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
  java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
  java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
  java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
  java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
  java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

Any clue for this? Thanks in advance!

[mpollmeier]

Your CPG has invalid METHOD node(s) - the error message indicates that it's missing an AST edge to a METHOD_RETURN node.

How did you create it, i.e. which frontend and/or command did you use (e.g. c2cpg)? Can you share the code you're scanning, or even better a minified subset?

[GiuliCler]

Thanks for the fast reply! I used the c2cpg script from command line (not importCode in Joern environment). I cannot share with you the code, but if it helps, in order to reduce the size of cpg generated (initial code was about 2G dimension) I removed some libraries in order to mantain only the core structure. Could it be the lack of reference to them that causes this problem?

[mpollmeier]

Could it be the lack of reference to them that causes this problem

I wouldn't think so, but really @max-leuthaeuser is the expert re c2cpg. Without some codebase to reproduce (as small as possible), this will be hard to debug and fix though.

[max-leuthaeuser]

@GiuliCler You could try loading the cpg into joern without running any passes to avoid the crash:

CpgLoader.loadFromOverflowDb(CpgLoaderConfig.withDefaults.withOverflowConfig(Config.withDefaults.withStorageLocation(cpgPath)))

You may be able to find the affected METHOD node (with the specific ID). Get its file, get its location, and try to reproduce a minimal example.

Thats pretty much what I would be doing to debug this but without the code or the cpg I cant do much here.

[GiuliCler]

Thank you both! I' ll try to apply the solution proposed by @max-leuthaeuser :)

[fabsx00]

Can we close this one?

[cyw3]

I have the same error:

Caused by: overflowdb.SchemaViolationException: OUT edge with label AST to an adjacent METHOD_RETURN is mandatory, but not defined for this METHOD node with id=134
        at io.shiftleft.codepropertygraph.generated.nodes.MethodDb.methodReturn(Method.scala:660)
        at io.shiftleft.codepropertygraph.generated.nodes.Method.methodReturn(Method.scala:286)
        at io.joern.x2cpg.passes.controlflow.cfgcreation.CfgCreator.<init>(CfgCreator.scala:55)
        at io.joern.x2cpg.passes.controlflow.CfgCreationPass.runOnPart(CfgCreationPass.scala:23)
        at io.joern.x2cpg.passes.controlflow.CfgCreationPass.runOnPart(CfgCreationPass.scala:17)
        at io.shiftleft.passes.ConcurrentWriterCpgPass.$anonfun$createApplySerializeAndStore$2(ParallelCpgPass.scala:252)
        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.util.NoSuchElementException: next on empty iterator
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:966)
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:964)
        at scala.collection.Iterator$$anon$7.next(Iterator.scala:528)
        at overflowdb.traversal.Traversal.next(Traversal.scala:22)
        at io.shiftleft.codepropertygraph.generated.nodes.MethodDb.methodReturn(Method.scala:655)
        ... 13 more

[max-leuthaeuser]

To debug this I need some piece of code that made trouble.

Get the METHOD node with id=134 and look at its name and file name. Are you able to reproduce this issue when generating a CPG for that particular method standalone?

[itsacoderepo]

The issue is inactive for a while, i close it for now. Please re-open it if the problem is still present.

[cyw3]

I have the same error:

Caused by: overflowdb.SchemaViolationException: OUT edge with label AST to an adjacent METHOD_RETURN is mandatory, but not defined for this METHOD node with id=134
        at io.shiftleft.codepropertygraph.generated.nodes.MethodDb.methodReturn(Method.scala:660)
        at io.shiftleft.codepropertygraph.generated.nodes.Method.methodReturn(Method.scala:286)
        at io.joern.x2cpg.passes.controlflow.cfgcreation.CfgCreator.<init>(CfgCreator.scala:55)
        at io.joern.x2cpg.passes.controlflow.CfgCreationPass.runOnPart(CfgCreationPass.scala:23)
        at io.joern.x2cpg.passes.controlflow.CfgCreationPass.runOnPart(CfgCreationPass.scala:17)
        at io.shiftleft.passes.ConcurrentWriterCpgPass.$anonfun$createApplySerializeAndStore$2(ParallelCpgPass.scala:252)
        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.util.NoSuchElementException: next on empty iterator
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:966)
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:964)
        at scala.collection.Iterator$$anon$7.next(Iterator.scala:528)
        at overflowdb.traversal.Traversal.next(Traversal.scala:22)
        at io.shiftleft.codepropertygraph.generated.nodes.MethodDb.methodReturn(Method.scala:655)
        ... 13 more

Hello, I still encounter this occasional exception frequently. Every time it occurs, the execution result of the tool will lack many Results that should originally exist, and the number of the log [INFO ] Enhancement io.joern.console.scan.ScanPass completed in 4450 ms. 130 + 650 changes committed from 25 parts. changed.

Here is a project that can be reproduced frequently (an average of 1 time in 4 executions), which can be tested based on this project. https://github.com/Haptic-Apps/Slide

The command line used is:

/data/test/joern-cli/joern-scan Slide/ --names=too-many-params --language=java --overwrite