Concurrent upsert causes deadlock. Is it normal?

[VasilyMelnik]

Hello everyone! I conduct small perftest on my local deplyment of ignite3, with 2 threads making 50% upsert and 50% get operations on random index:

public static void main(String[] args) throws Exception
    {
        String[] addresses = {
                "localhost:10800",
                "localhost:10801",
                "localhost:10802"
        };       
        try (IgniteClient client = IgniteClient.builder()
                .addresses(addresses)
                .connectTimeout(30000)
                .build()) {

            Runnable runnable = new Runnable() {
                @Override
                public void run() {
                    Random randomOp = new Random();
                    Random randomIndex = new Random();
                    Table perftest =client.tables().table("perftest");
                    long start = System.currentTimeMillis();
                    int N = 70000;
                    for (int i=0;i<N; i++) {
                        // 50% upsert, 50% get
                        if (randomOp.nextInt(10) <5 )
                                perftest.recordView().upsert(null, Tuple.create().set("id", randomIndex.nextLong(1000)).set("val", RandGeneratedStr(1000)));
                            else
                                perftest.recordView().get(null,Tuple.create().set("id", randomIndex.nextLong(1000)));
                    }
                    long end =System.currentTimeMillis();
                    System.out.println("Rate: "+N*1.0*1000/(end-start));
                }
            };
          Thread t1 = new Thread(runnable);
          Thread t2 = new Thread(runnable);

          t1.start();
          t2.start();

          t1.join();
          t2.join();
        }
    }

And when i run two concurrent threads, i get one of them failed with: IGN-TX-4 TraceId:205ebf31-614f-4ab1-bb3c-c017ab57ec09 Failed to acquire a lock due to a possible deadlock I see, that each upsert requires X lock on key (smth like this LockKey [ctx=42_part_11, key=RowId [partitionId=11, uuid=00000199-7235-8748-3904-4e30efc61428]]) and get requires S lock on key. All upserts by one key should be serialized? getiing X-lock one by one. So what is the reason of deadlock and how can i prevent it in concurrent workloads?

[ptupitsyn]

"Failed to acquire a lock due to a possible deadlock" is expected in concurrent workloads and should be retried. Upcoming Ignite 3.1 will provide runInTransaction method with automatic reries:

https://github.com/apache/ignite-3/blob/main/modules/api/src/main/java/org/apache/ignite/tx/RunInTransactionInternalImpl.java

[VasilyMelnik]

@ptupitsyn Pavel, thank you. But i still do not understand, what is the reason of deadlocks? Each implicit transaction within upsert operation aquires only one lock on one key? as far as i undesrtand. Where is deadlock situation?

[ptupitsyn]

It is not a deadlock, just a concurrent update, maybe the message is a bit misleading.

[VasilyMelnik]

@ptupitsyn So ignite3 will never support real pessimistic lock for updates in SQL Server/Oracle manner and we should use retry? No way to implement SELECT FOR UPDATE + UPDATE scenario in one transaction with row locking and no retries?

[ptupitsyn]

Can't say for sure about the future, but it seems to be the state of things at the moment.

[ascherbakoff]

Hello. Implicit transactions doesn't require manual retry, but this is not implemented in the current release. This will be improved in 3.2.