awslabs
Heartbeat failure
Hi, We are trying to use https://github.com/awslabs/dynamodb-lock-client to do leader election and doing some test.
Our test is super simple and what we do is like: client = new AmazonDynamoDBLockClient( AmazonDynamoDBLockClientOptions.builder(dynamoDB, config.tableName()) .withPartitionKeyName(config.partitionKey()) .withTimeUnit(TimeUnit.MILLISECONDS) .withLeaseDuration(1000) .withHeartbeatPeriod(100) .withOwnerName("test") .withCreateHeartbeatBackgroundThread(true) .build()); lockItem = client.tryAcquireLock( AcquireLockOptions .builder(config.lockValue()) .withDeleteLockOnRelease(false) .withAdditionalTimeToWaitForLock(300) .withTimeUnit(TimeUnit.MILLISECONDS) .withRefreshPeriod(100) .build());
while(true) { Optional<LockItem> existing = client.getLock(config.lockValue(), Optional.empty()); Thread.sleep(300); }
And we test above programs by just using 1 single server. the heartbeat is using 100ms, and when program started, it works well, get the lock, and every 100ms the heartbeat thread will update the version number . But randomly after 2 or 3 minutes, problems happened, by enabling the logs, I found the function sendHeartBeat, will throw ConditionalCheckFailedException, since it is trying to conditionally update the record, but the question is there is only single server running this, and I can not find another places trying to update the record concurrently, so what's going on here?
