Table Locking issue with SQLG

[amitbhoyar]

Hi Pieter, Hope you're doing well. I am impressed with your work on SQLG apache Tinkerpop implementation. I am working on a small project on a graph database with SQLG. where I am facing the table locking issue with the DML. I am using sqlg version: 2.1.6 sqlg-postgres-dialect version : 2.1.6.

scenario 1: whenever I am trying to create a link with two vertices for the first time I am facing a table lock issue (fig1). SQL from log : ALTER TABLE "icedq_catalog"."E_Dependency" ADD COLUMN "icedq_catalog.Column.id__O" TEXT I have attached my connection file. Your little help will save my life. Thanks in advance.

[amitbhoyar]

package org.icedq.datacatalog.api.v1.configuration;

import lombok.extern.slf4j.Slf4j; import org.apache.commons.configuration2.PropertiesConfiguration; import org.icedq.commons.errorcodes.ApplicationErrorCode; import org.icedq.datacatalog.api.v1.configuration.properties.SqlgProperties; import org.icedq.datacatalog.api.v1.exception.ApplicationException; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Scope; import org.umlg.sqlg.structure.SqlgGraph; import org.umlg.sqlg.structure.topology.Schema;

import java.util.Optional;

@Slf4j @Configuration @Scope("singleton") public class SqlgGraphConfiguration {

private SqlgGraph sqlgGraph; @Autowired SqlgProperties sqlgProperties;

@Bean public SqlgGraph initialize() { if (sqlgGraph == null) { PropertiesConfiguration properties = new PropertiesConfiguration(); properties.setProperty("jdbc.url", sqlgProperties.getUrl()); properties.setProperty("jdbc.username", sqlgProperties.getUsername()); properties.setProperty("jdbc.password", sqlgProperties.getPassword()); try { sqlgGraph = SqlgGraph.open(properties); } catch (IllegalArgumentException e) { log.error("Error while Initializing the SQLG::" + e.getMessage()); } } return null; }

public SqlgGraph getSqlgGraph() { return sqlgGraph; }

public Schema getSchemaInstance() { Optional<Schema> schema = sqlgGraph.getTopology().getSchema(sqlgProperties.getSchema()); if (schema.isPresent()) { return schema.get(); } else { throw new ApplicationException( sqlgProperties.getSchema() + " not found", ApplicationErrorCode.UNEXPECTED_ERROR); } } }

[pietermartin]

Can you show the code that does the topology creation? Either by Sqlg's ensureXXXExists methods or by adding vertices or edges in which case Sqlg will create the schema automatically.

What is the purpose of your getSchemaInstance() method?

Further, 99% of locks on Postgresql is created by transactions that do not call commit or rollback. It is the responsibility of the client/user to call commit or rollback on the transaction. i.e graph.txt().commit/rollback()

[amitbhoyar]

here is the code

try { Edge edge = vertexOut.addEdge(edgeLabel.getLabel(), vertexIn, properties.toArray()); sqlgGraphConfiguration.getSqlgGraph().tx().commit(); return edge; } catch (Exception e) { e.printStackTrace(); return null; }

[amitbhoyar]

I am first creating Edge Label with say dummy Vertex Lable Dummy. Then I am trying to add a link with the Edge edge = vertexOut.addEdge(edgeLabel.getLabel(), vertexIn, properties.toArray());

where outvertex and inVertex are already Created.

[pietermartin]

Ok, so the catch part of the exception should call rollback() on the transaction. Every thread should always end with either commit or rollback otherwise you will risk having idle in transaction session on posrtgresql. In general, NEVER swallow exceptions, it causes nothing but pain.

You can use the following query to see which transaction/query is blocking which transaction/query.

select pid, 
       usename, 
       pg_blocking_pids(pid) as blocked_by, 
       query as blocked_query
from pg_stat_activity
where cardinality(pg_blocking_pids(pid)) > 0;

Or use pgadmin4, it has a dashboard that displays the blocking queries including what blocks what.

[amitbhoyar]

Thanks, I will handle all the Exceptions But I am not getting any exceptions in this scenario

[pietermartin]

In the DBeaver screen shot there are multiple active sql statements, including two ALTER TABLE statements, ALTER statements take a table level lock on postgresql. However postgresql will throw an exception if there are dead locks. If its hanging then its because there are connections that are not committing or rolling back, forcing other connections to wait.

[amitbhoyar]

Got it. I will check with the commit and Rollback. Do I also explicitly need to do this for Read-only Operations? like getVertex, get edge, traversal ?

[pietermartin]

Yes, all transactions. This is both a TinkerPop feature and a postgresql feature. TinkerPop automatically starts a transaction and for postgresql every sql query runs in a transaction.

I see you are using Spring, so I'd suggest somewhere Spring will have some interceptor where you can ensure that every thread will call rollback when it completes. This will ensure that some other developer does not accidentally leave transaction open.

[pietermartin]

Some more advice, if your application is going to create many topology elements in parallel, there is always a risk that postgresql might throw a dead lock exception. To prevent the risk of this its best to know where in your application you need to create new topology elements and use Sqlgs ensureXXXExists() methods to create the topology upfront.

[amitbhoyar]

Thanks a lot for the quick response and guidance. I will work on it and let you know if I still face this issue.

[pietermartin]

Feel free to reopen this.