Synchronisation error

[ralfeus]

Describe the bug After some moment, probably after the key is expired the errors are logged every 0.15 seconds:

SEVERE|2022-10-06 15:22:04.682472|SyncService|exception syncing entry to local {atKey: @baboon9blue:memberinviterequest.atkrypto@81bored7, value: null, metadata: null, commitId: 125, operation: -} - Failed to decryptData caused by
Decryption failed. Encrypted value is null 

SEVERE|2022-10-06 15:22:04.846217|SyncService|exception syncing entry to local {atKey: @baboon9blue:memberinviterequest.atkrypto@81bored7, value: null, metadata: null, commitId: 125, operation: -} - Failed to decryptData caused by
Decryption failed. Encrypted value is null 

When the key with same name is created again it's never synchronised to the remote secondary.

To Reproduce Steps to reproduce the behavior:

1. Receive a key with TTL 60 seconds. The code to send a key:

  final Metadata _defaultMetadata = Metadata( 👍 
    ..isEncrypted = true
    ..isPublic = false
    ..namespaceAware = true
    ..ttl = 60000
    ..ttr = -1;
    
    Future<bool> send(String recipient, String key, dynamic value) async {
    var atKey = AtKey()
      ..key = key
      ..sharedBy = client.getCurrentAtSign()
      ..sharedWith = recipient
      ..metadata = _defaultMetadata;
    Logger.root.finest(atKey.key);
    final result = await client.put(atKey, value);
    return result;
  }

2. Have the recipient to get the key. The receiving code:

  void listen() async {
    final atStream =
        AtClientManager.getInstance().notificationService.subscribe();
    await for (final notification in atStream) {
      // whatever
    }
  }

3. Wait the key to expire.
4. Observe errors above

Expected behavior No errors upon key's expiration

[gkc]

Thanks @ralfeus - an unexpected consequence of some fairly recent changes. We'll fix the bug and add some tests to cover this scenario.

[ralfeus]

Not sure if it's related or a different issue - please advice. I put a key memberInviteRequest shared with @81bored7 to a store. Then it's supposed to be synchronised and I see following in the log:

I/flutter (20529): INFO|2022-10-10 00:25:10.028272|AtLookup|Creating new connection
I/flutter (20529): INFO|2022-10-10 00:25:10.856429|AtLookup|auth success
I/flutter (20529): INFO|2022-10-10 00:25:11.009495|SyncService|Returning the serverCommitId 273
I/flutter (20529): INFO|2022-10-10 00:25:11.168953|SyncService|Returning the serverCommitId 273
I/flutter (20529): SEVERE|2022-10-10 00:25:11.538701|SyncService|@81bored7:memberinviterequest.atkrypto@baboon9blue is not found in keystore. Skipping to entry to sync

Naturally nothing is synchronised.

[murali-shris]

@ralfeus Thanks for sharing the logs. I am trying to replicate your issue in my local environment. I will keep you updated with my findings.

[murali-shris]

Updates from my local testing:

1. there is discrepancy in the way keys are added in metadata cache in create and put methods of hive_keystore.in put method we are not encoding the key before adding to cache. whereas in create method we are encoding the key.
2. Due to issue # 1 certain expired keys still remain in cache since remove(key) doesn't clear the entry from cache.
3. Side effects of issue # 1. Same keys gets deleted again and again and increments commit log. Repeated sync of same key deletion happen.

Proposed fix:

• encode the keys in put method and then add to cache
• revisit hive keystore for any other encode inconsistencies
• while removing from cache, have two operations - a) to delete unencoded key(due to issue # 1) and b) delete encoded key

[murali-shris]

@ralfeus do you have local secondary server setup where you can run your client pointing to local server?

[ralfeus]

No, I don't use local secondary server.

On Tue, Oct 11, 2022 at 4:07 PM Muralidharan @.***> wrote:

@ralfeus https://github.com/ralfeus do you have local secondary server setup where you can run your client pointing to local server?

— Reply to this email directly, view it on GitHub https://github.com/atsign-foundation/at_client_sdk/issues/742#issuecomment-1274748490, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPDQ3YI3DHYZECDIHVDTNLWCVYBRANCNFSM6AAAAAAQ6XZG2Q . You are receiving this because you were mentioned.Message ID: @.***>

[murali-shris]

pending addressing some review comments and PR merge.

[murali-shris]

Discussion from architecture call. While deleting non existent keys, throw KeyNotFoundException instead of data:null. Review the impact of changes on app/client side.

[murali-shris]

Updated server pull request https://github.com/atsign-foundation/at_server/pull/966 with latest changes for throwing KeyNotFoundException. Working on changes in at_client

[murali-shris]

https://github.com/atsign-foundation/at_client_sdk/pull/768/files client PR review comments addressed. Pending final review and merge

[murali-shris]

https://github.com/atsign-foundation/at_client_sdk/pull/768/files client PR review comments addressed. Pending final review and merge

Test scenarios covered test 1: put a key using at_client and then delete the key. Delete should return true test 2: put a key at_client, delete the key. Now try deleting the same key again. First delete should return true, second delete should return false.

[murali-shris]

@kalluriramkumar please update this ticket once we have the fix for infinite loop issue on sync service published.

[sitaram-kalluri]

The following PR fixes the issue: https://github.com/atsign-foundation/at_client_sdk/pull/814

[sitaram-kalluri]

The changes are published in at_client v3.0.45

[murali-shris]

@ralfeus can you please update to at_client v3.0.45 and check whether your issue is resolved

[gkc]

Please note that other changes you need will be in 3.0.46 (expect to publish tomorrow)