Freeze on offline WiFi with Selective Sync: Query and Save calls get stuck in await

[giulitu95]

Description

This problem occurs on Android platforms when the device is connected to a wify network without internet connection. If a selective sync is configured and data are cyclically saved on the datastore, eventually, the execution gets stuck in the save method. My configuration includes Datastore, Api and Auth plugins.

Categories

• [ ] Analytics
• [ ] API (REST)
• [ ] API (GraphQL)
• [ ] Auth
• [ ] Authenticator
• [X] DataStore
• [ ] Notifications (Push)
• [ ] Storage

Steps to Reproduce

1. Start the app with internet connection
2. Initialize amplify
3. Sign-up and Sign-in with Cognito authentication
4. Switch the device to a wify without internet connection
5. Cyclically call the method await Amplify.Datastore.save(model)
6. Eventually, the execution gets stuck in an await

Screenshots

No response

Platforms

• [ ] iOS
• [X] Android
• [ ] Web
• [ ] macOS
• [ ] Windows
• [ ] Linux

Flutter Version

3.24.4

Amplify Flutter Version

2.5.0

Deployment Method

Amplify CLI (Gen 1)

Schema

No response

[Equartey]

Hi @giulitu95, thanks for taking the time to open this issue. We will investigate this issue and provide an update when we can.

[ekjotmultani]

Hello @giulitu95, I've been unable to reproduce this issue. Are you using a physical device to get this issue? Also, are the cyclical calls to save being made programmatically or by repeated user input?

[giulitu95]

Hi @ekjotmultani, I'm using a physical device (Samsung Galaxy S22). I have been able to reproduce the problem on a simple app calling the Amplify.DataStore.save() method in an infinite loop programmatically.

final mv = MachineVersion(machineSerialNumber: 23);
while (true) {
  print("-- Saving");
  try {
    await Amplify.DataStore.save(mv);
  } catch (e) {
    print("-- Error: $e");
  }
  print("-- Saved");;
}

After some cycles, the program gets stucks printing as last message "--Saving" and the execution never enters in the catch block. This occurs only when the device has not internet connection but is connected to a local WiFi network (without internet connection). The following is my amplify configuration:

final auth = AmplifyAuthCognito();
final api = AmplifyAPI();
final datastore = AmplifyDataStore(
    modelProvider: ModelProvider.instance,
    options: DataStorePluginOptions(syncExpressions: [
      DataStoreSyncExpression(
          MachineVersion.classType, () => MachineVersion.ID.eq("test")),
    ]));
await Amplify.addPlugins([datastore, api, auth]);
await Amplify.configure(amplifyconfig);

If I do not specify a DataStoreSyncExpression (removing the entire options attribute), the execution does not stuck.

This is the graphQl for the model used as example (but it happens with any models defined in my graphQl):

[...]
type MachineVersion @model @auth(
        rules: [
            { allow: groups, groups: ["Admins"], operations: [create, read, update, delete]}
            { allow: owner }
        ]
    ){
    id: ID! @primaryKey
    machineSerialNumber: Int!
    remoteSoftwareUpdateDate: String
    workingData: [WorkingData] @hasMany(fields: ["id"])
    diagnosisSession: [DiagnosisSession] @hasMany(fields: ["id"])
    machineComponent: [MachineComponent] @hasMany(fields: ["id"])
    owner: String @auth(
        rules: [
            { allow: groups, groups: ["Admins"], operations: [create, read, update, delete]}
            { allow: owner, operations: [read, delete] }
        ]
    )
}
[...]

Note also that whenever Amplify tries to save the model (before it gets stuck), the debug console shows many exceptions as:

E/amplify:aws-datastore( 4591): Failure encountered while attempting to start API sync.
E/amplify:aws-datastore( 4591): DataStoreException{message=DataStore subscriptionProcessor failed to start., cause=DataStoreException{message=Error during subscription., cause=ApiException{message=Connection failed., cause=null, recoverySuggestion=Sorry, we don’t have a recovery suggestion for this error.}, recoverySuggestion=Evaluate details.}, recoverySuggestion=Check your internet.}
E/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$4$com-amplifyframework-datastore-syncengine-Orchestrator(Orchestrator.java:344)
E/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.syncengine.Orchestrator$$ExternalSyntheticLambda8.subscribe(Unknown Source:2)
[...]
E/amplify:aws-datastore( 4591): Caused by: DataStoreException{message=Error during subscription., cause=ApiException{message=Connection failed., cause=null, recoverySuggestion=Sorry, we don’t have a recovery suggestion for this error.}, recoverySuggestion=Evaluate details.}
E/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$subscription$2(AppSyncClient.java:322)
E/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.appsync.AppSyncClient$$ExternalSyntheticLambda5.accept(Unknown Source:4)
[...]
E/amplify:aws-datastore( 4591): Caused by: ApiException{message=Connection failed., cause=null, recoverySuggestion=Sorry, we don’t have a recovery suggestion for this error.}
E/amplify:aws-datastore( 4591): 	at com.amplifyframework.api.aws.SubscriptionEndpoint.requestSubscription(SubscriptionEndpoint.java:152)
E/amplify:aws-datastore( 4591): 	... 7 more

[...]

W/amplify:aws-datastore( 4591): API sync failed - transitioning to LOCAL_ONLY.
W/amplify:aws-datastore( 4591): DataStoreException{message=Error during subscription., cause=ApiException{message=Thread interrupted waiting for connection acknowledgement, cause=null, recoverySuggestion=Sorry, we don’t have a recovery suggestion for this error.}, recoverySuggestion=Evaluate details.}
W/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$subscription$2(AppSyncClient.java:322)
W/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.appsync.AppSyncClient$$ExternalSyntheticLambda5.accept(Unknown Source:4)
[...]
W/amplify:aws-datastore( 4591): Caused by: ApiException{message=Thread interrupted waiting for connection acknowledgement, cause=null, recoverySuggestion=Sorry, we don’t have a recovery suggestion for this error.}
W/amplify:aws-datastore( 4591): 	at com.amplifyframework.api.aws.SubscriptionEndpoint.requestSubscription(SubscriptionEndpoint.java:152)
W/amplify:aws-datastore( 4591): 	... 7 more

[...]

W/amplify:aws-datastore( 4591): DataStore failed to start after emitter was disposed.
W/amplify:aws-datastore( 4591): DataStoreException{message=Subscription operations were interrupted during setup., cause=null, recoverySuggestion=Retry}
W/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.syncengine.SubscriptionProcessor.startSubscriptions(SubscriptionProcessor.java:155)
W/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$4$com-amplifyframework-datastore-syncengine-Orchestrator(Orchestrator.java:342)

[...]

W/amplify:aws-datastore( 4591): DataStoreException{message=Error during subscription., cause=ApiException{message=Interrupted waiting for Cognito Userpools token., cause=java.lang.InterruptedException, recoverySuggestion=Sorry, we don’t have a recovery suggestion for this error.}, recoverySuggestion=Evaluate details.}
W/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$subscription$2(AppSyncClient.java:322)
W/amplify:aws-datastore( 4591): 	at com.amplifyframework.datastore.appsync.AppSyncClient$$ExternalSyntheticLambda5.accept(Unknown Source:4)
[...]
W/amplify:aws-datastore( 4591): Caused by: ApiException{message=Interrupted waiting for Cognito Userpools token., cause=java.lang.InterruptedException, recoverySuggestion=Sorry, we don’t have a recovery suggestion for this error.}
W/amplify:aws-datastore( 4591): 	at com.amplifyframework.api.aws.sigv4.DefaultCognitoUserPoolsAuthProvider.fetchToken(DefaultCognitoUserPoolsAuthProvider.java:73)
W/amplify:aws-datastore( 4591): 	at com.amplifyframework.api.aws.sigv4.DefaultCognitoUserPoolsAuthProvider.getLatestAuthToken(DefaultCognitoUserPoolsAuthProvider.java:87)
[...]
W/amplify:aws-datastore( 4591): Caused by: java.lang.InterruptedException
W/amplify:aws-datastore( 4591): 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1048)
W/amplify:aws-datastore( 4591): 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:318)

Also other exception occurs, let me know if you need the entire detailed debug log

[NikaHsn]

@giulitu95 thank you for providing these details. we will look into this issue and get back to you with any updates.

[giulitu95]

Thank you @NikaHsn. I forgot to mention that the problem is not related only to the save method, but if I alternate save with query methods the execution could get stuck also in the query future.

[NikaHsn]

@giulitu95 thanks for providing more details on this. we will look into this issue and provide update as we have them.

[ekjotmultani]

@giulitu95 thanks again for the added information! I have been successful in reproducing this issue and we will track it as a bug. I also noticed the massive amounts of other exceptions in the output so we'll investigate that too

[giulitu95]

Is there any update on this issue? This bug makes it impossible to use selective sync, and I was wondering if there are any plans to address it. Thanks!

[ekjotmultani]

Hi @giulitu95, unfortunately we have no updates yet, we are still investigating the issue however. A suspected root cause being how network connection is checked on android when datastore attempts to save, I will of course keep you updated when we do find a cause and remedy it

[giulitu95]

Any update of this?

[tyllark]

Hello @giulitu95, thanks for nudge on the issue. @ekjotmultani is looking into the issue and will provide any updates here.