SQL: Out-of-memory when `CREATE EDGE ... IF NOT EXISTS` for existing edges with array of destination vertices

[gramian]

ArcadeDB Version:

ArcadeDB Server v24.11.1-SNAPSHOT (build 4207821d055e37cbd74b76683e45073da759c2de/1729083296142/console)

OS and JDK Version:

Running on Mac OS X 12.7.6 - OpenJDK 64-Bit Server VM 17.0.12 (Homebrew)

When creating edges that already exists with the suffix IF NOT EXISTS and the destinition (TO part) is an array, then an out-of-memory error happens after some time.

Expected behavior

No out-of-memory (OOM) error.

Actual behavior

Error on command execution (PostCommandHandler)
java.lang.OutOfMemoryError: Java heap space

or:

Error on command execution (PostCommandHandler)
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3481)
	at java.base/java.util.ArrayList.grow(ArrayList.java:237)
	at java.base/java.util.ArrayList.grow(ArrayList.java:244)
	at java.base/java.util.ArrayList.add(ArrayList.java:454)
	at java.base/java.util.ArrayList.add(ArrayList.java:467)
	at com.arcadedb.query.sql.executor.InsertExecutionPlan.executeInternal(InsertExecutionPlan.java:67)
	at com.arcadedb.query.sql.executor.ScriptLineStep.syncPull(ScriptLineStep.java:48)
	at com.arcadedb.query.sql.executor.ScriptExecutionPlan.doExecute(ScriptExecutionPlan.java:101)
	at com.arcadedb.query.sql.executor.ScriptExecutionPlan.fetchNext(ScriptExecutionPlan.java:62)
	at com.arcadedb.query.sql.parser.LocalResultSet.fetchNext(LocalResultSet.java:44)
	at com.arcadedb.query.sql.parser.LocalResultSet.<init>(LocalResultSet.java:40)
	at com.arcadedb.query.sql.SQLScriptQueryEngine.executeInternal(SQLScriptQueryEngine.java:218)
	at com.arcadedb.query.sql.SQLScriptQueryEngine.command(SQLScriptQueryEngine.java:108)
	at com.arcadedb.database.LocalDatabase.command(LocalDatabase.java:1348)
	at com.arcadedb.server.ServerDatabase.command(ServerDatabase.java:472)
	at com.arcadedb.server.http.handler.PostCommandHandler.executeCommand(PostCommandHandler.java:134)
	at com.arcadedb.server.http.handler.PostCommandHandler.execute(PostCommandHandler.java:113)
	at com.arcadedb.server.http.handler.DatabaseAbstractHandler.execute(DatabaseAbstractHandler.java:100)
	at com.arcadedb.server.http.handler.AbstractServerHttpHandler.handleRequest(AbstractServerHttpHandler.java:127)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:395)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.base/java.lang.Thread.run(Thread.java:840)

Steps to reproduce

1. Create vertex and edge types

CREATE VERTEX TYPE vex;
CREATE EDGE TYPE edg;

2. Add some vertices

CREATE VERTEX vex;
CREATE VERTEX vex;
CREATE VERTEX vex;

3. Add new edges

CREATE EDGE edg FROM #1:0 TO [#4:0,#7:0]

4. Try add same edges again with IF NOT EXISTS

CREATE EDGE edg FROM #1:0 TO [#4:0,#7:0] IF NOT EXISTS

This last statement causes the OOM error.

[lvca]

That's pretty awful, I'll look it up asap.

[gramian]

Some more details and a more reduced test case:

• This happens only if the source FROM ... or the target TO ... is an array of more than one element.
• Another test case:

CREATE VERTEX TYPE vex;
CREATE EDGE TYPE edg;
CREATE VERTEX vex;

CREATE EDGE edg FROM #1:0 TO #1:0;
CREATE EDGE edg FROM [#1:0,#1:0] TO [#1:0,#1:0] IF NOT EXISTS;

BTW: Using CREATE EDGE edg FROM #1:0 TO [#1:0,#1:0] IF NOT EXISTS first, creates two edges; is this correct?

[robfrank]

I was able to reproduce the problem on my side. Stay tuned

[robfrank]

Reworked and merged. Closing.

[gramian]

@robfrank I am testing the latest head with the fix. The OOM is gone, but it seems the IF NOT EXISTS is somewhat ignored. For example the following commands keep on creating the same edges:

CREATE VERTEX TYPE vex;
CREATE EDGE TYPE edg;
CREATE VERTEX vex;
CREATE VERTEX vex;
CREATE VERTEX vex;

CREATE EDGE edg FROM #1:0 TO [#1:1,#1:2];
CREATE EDGE edg FROM #1:0 TO [#1:1,#1:2] IF NOT EXISTS;
CREATE EDGE edg FROM #1:0 TO [#1:1,#1:2] IF NOT EXISTS;
...

[robfrank]

It returns the existing ones . Underneath is very hard to skip the existing ones, so we took the decision to return new created edges with previous in the same result set

[gramian]

I will test again, but in the graph view it looked like each CREATE EDGE creates new edges, which should not happen for IF NOT EXISTS, if I am not wrong.

[robfrank]

do you mean, via studio?

[gramian]

Yes. I tested the changes manually in studio with the example above. Everytime I call CREATE EDGE ... IF NOT EXISTS it seems new edges are created as new edges between the same vertices appear in the graph view.

[robfrank]

Let me talk with @lvca to find a proper solution. If you query the database, you should find the right amount of edges, this is what happens in the unit tests. The best solution would be to return only the newly created edges, but it is very tricky to implement that.

[gramian]

IMHO the problem is not what is getting returned, but what is created: There should not be any new edges created if edges between the vertices already exists.

Sorry if I misunderstand you.

[gramian]

@robfrank The latest changes fixed the problem. Great! Thanks