Error with pessimistic transactions

[tobiemh]

When using pessimistic transactions, the following issues arise:

1. Running multiple successful, sequential, pessimistic transactions inside a single application invocation works fine.
2. Running the invocation for a second time, once the first invocation has finished, results in Error: PessimisticLockError { inner: ResolveLockError, success_keys: [] } errors.
3. TiKV is left in an incorrect state, with locks on the keys.
4. All subsequent invocations fail when attempting to write to these same keys (even though there are no other clients running at the same time).

To reproduce this error, run the sample code below. You will see that:

1. tiup playground --tag surrealdb --mode tikv-slim --pd 1 --kv 1
2. Running this example app with cargo run works fine.
3. Running this example a second time with cargo run results in errors.
4. Running this example a third time with cargo run results in errors.
5. ...

This has the same result when using version 0.1.0 and/or the current master branch.

Cargo.toml

[package]
name = "tikv-test"
edition = "2021"
version = "0.0.0"

[dependencies]
tikv-client = { version = "0.1.0", git = "https://github.com/tikv/client-rust" }
tokio = { version = "1.20.1", features = ["macros", "rt-multi-thread"] }

src/main.rs

use tikv_client::CheckLevel;
use tikv_client::TransactionOptions;
use tikv_client::TransactionClient;

#[tokio::main]
async fn main() -> Result<(), tikv_client::Error> {
	
	let ds = TransactionClient::new(vec!["127.0.0.1:2379"], None).await?;
	let op = TransactionOptions::new_pessimistic().drop_check(CheckLevel::Warn);

	// Start a new transaction

	let mut tx = ds.begin_with_options(op.clone()).await?;
	tx.put(b"one".to_vec(), b"one".to_vec()).await?;
	println!("Line 14 ok");
	tx.put(b"two".to_vec(), b"two".to_vec()).await?;
	println!("Line 16 ok");
	tx.commit().await?;
	println!("Line 18 ok");

	// Start a new transaction

	let mut tx = ds.begin_with_options(op.clone()).await?;
	tx.put(b"one".to_vec(), b"one".to_vec()).await?;
	println!("Line 24 ok");
	tx.put(b"two".to_vec(), b"two".to_vec()).await?;
	println!("Line 26 ok");
	tx.commit().await?;
	println!("Line 28 ok");

	// Start a new transaction

	let mut tx = ds.begin_with_options(op.clone()).await?;
	tx.put(b"one".to_vec(), b"one".to_vec()).await?;
	println!("Line 34 ok");
	tx.put(b"two".to_vec(), b"two".to_vec()).await?;
	println!("Line 36 ok");
	tx.commit().await?;
	println!("Line 38 ok");

    Ok(())
}

Example output the first time is as follows:

% cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.33s
     Running `target/debug/tikv-test`
Aug 30 21:20:14.210 INFO connect to tikv endpoint: "127.0.0.1:20160"
Line 14 ok
Line 16 ok
Line 18 ok
Line 24 ok
Line 26 ok
Line 28 ok
Line 34 ok
Line 36 ok
Line 38 ok

Example output the second time is as follows:

% cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/tikv-test`
Aug 30 21:20:16.738 INFO connect to tikv endpoint: "127.0.0.1:20160"
Line 14 ok
Aug 30 21:20:17.744 WARN Dropping an active transaction. Consider commit or rollback it., child: 1
Error: PessimisticLockError { inner: ResolveLockError, success_keys: [] }

[ekexium]

Thanks for the report! I think it's something wrong with the backoff mechanism. By manually setting backoff strategies like this it works fine for me.

    let options = RetryOptions {
        region_backoff: Backoff::no_jitter_backoff(2, 500, 10),
        lock_backoff: Backoff::no_jitter_backoff(2, 500, 10),
    };
    let mut op = TransactionOptions::new_pessimistic()
        .drop_check(CheckLevel::Warn)
        .retry_options(options);