Datastore sync timing issue on poor network connection

[JncHrmn]

Description

Hi there,

We need guidance on how to reliably coordinate Amplify DataStore synchronization with our app’s navigation flow. We’ve observed two problematic scenarios:

1. App cold start (user already signed in):

• On launch, we show the native splash screen until the DataStore emits its ready event, then navigate to the Home screen and immediately run our DataStore queries.

• Occasionally, despite receiving ready, our initial queries return empty lists. Manually pulling to refresh then populates the data, suggesting we’re querying too early.

Our current approach for the cold start:

bool _userWasSignedInAtLaunch = false;

void main() async {
  WidgetsBinding widgetsBinding = WidgetsFlutterBinding.ensureInitialized();
  FlutterNativeSplash.preserve(widgetsBinding: widgetsBinding);

  await _configureAmplify();

  final session = await Amplify.Auth.fetchAuthSession(
    options: const FetchAuthSessionOptions(forceRefresh: false),
  );
  _userWasSignedInAtLaunch = session.isSignedIn;

  if (!_userWasSignedInAtLaunch) {
    FlutterNativeSplash.remove();
  }

  runApp(
    MyApp(),
  );
}

Future<void> _configureAmplify() async {
  try {
    final auth = AmplifyAuthCognito();
    final datastorePlugin =
        AmplifyDataStore(modelProvider: ModelProvider.instance);
    final api = AmplifyAPI(
      options: APIPluginOptions(modelProvider: ModelProvider.instance),
    );
    final storage = AmplifyStorageS3();
    if (Amplify.isConfigured == false) {
      await Amplify.addPlugins([datastorePlugin, auth, api, storage]);
    }
    await Amplify.configure(amplifyconfig);

    Amplify.Hub.listen(HubChannel.DataStore, (hubEvent) {
      if (hubEvent.eventName == 'ready') {
        if (_userWasSignedInAtLaunch) {
          FlutterNativeSplash.remove();
        }
      }
    });

    Amplify.Hub.listen(HubChannel.Auth, (hubEvent) async {
      switch (hubEvent.type) {
        case AuthHubEventType.signedIn:
          safePrint('User is signed in.');
          goRouter.go('/splash');

          break;
        case AuthHubEventType.signedOut:
          safePrint('User is signed out.');
          break;
        case AuthHubEventType.sessionExpired:
          safePrint('The session has expired.');
          break;
        case AuthHubEventType.userDeleted:
          safePrint('The user has been deleted.');
          break;
      }
    });
    safePrint("Amplify has been configured successfully.");
  } on Exception catch (e) {
    safePrint('An error occurred configuring Amplify: $e');
  }
}

2. Login flow (user signed out → signed in) on unreliable networks:

• Immediately after a successful sign-in, we route to a loading screen and kick off a full DataStore sync. We deliberately keep the user stuck on that screen until sync completes, because we don’t want to show an empty Home.

• Problem: On a poor connection, the sync can stall indefinitely, trapping the user on the loading screen. The only workaround currently is to restart the app—since they’re still authenticated, the app jumps straight to Home, but then shows a completely empty UI even though data exists in the cloud.

• This exposes that our “wait-for-ready” approach isn’t robust: neither retrying nor back-off logic is in place, and no fallback path exists to recover when sync never finishes.

Listen to the signedIn Event

 Amplify.Hub.listen(HubChannel.Auth, (hubEvent) async {
      switch (hubEvent.type) {
        case AuthHubEventType.signedIn:
          safePrint('User is signed in.');
          goRouter.go('/loadingScreen');

          break;

Within the loading widget/screen the trigger the following:

class DataStoreSyncService {
  Future<void> syncAfterSignIn() async {
    final completer = Completer<void>();

    final sub = Amplify.Hub.listen(HubChannel.DataStore, (hubEvent) {
      if ((hubEvent.eventName == 'ready') && !completer.isCompleted) {
        completer.complete();
      }
    });

    await Amplify.DataStore.clear();
    await Amplify.DataStore.start();

    await completer.future;
    await sub.cancel();
  }
}

What’s the recommended way to detect and wait for a truly complete initial sync, beyond listening for the ready event? How can we implement retry/back-off or timeout logic so that users aren’t stranded on a loading screen forever? Can we leverage additional Hub events (e.g. network status changes) or DataStore APIs (like observeQuery().isSynced) to build a more resilient loading flow and fallback UI?

We’ve tried following the Amplify docs examples, but still run into this race condition on slow networks. Any pointers, sample patterns, or code snippets would be greatly appreciated. Thanks!

Categories

• [ ] Analytics
• [ ] API (REST)
• [ ] API (GraphQL)
• [ ] Auth
• [ ] Authenticator
• [x] DataStore
• [ ] Notifications (Push)
• [ ] Storage

Steps to Reproduce

1. Start App + DataStore
2. Wait until DataStore is ready
3. Navigate to homescreen and fetch data

Or.

1. Login Screen -> Login
2. Switch to loading to sync DataStore and wait until ready
3. Navigate to homescreen and fetch data

Screenshots

No response

Platforms

• [x] iOS
• [x] Android
• [ ] Web
• [ ] macOS
• [ ] Windows
• [ ] Linux

Flutter Version

3.29.3

Amplify Flutter Version

2.6.0

Deployment Method

Amplify CLI (Gen 1)

Schema

[tyllark]

Hello @JncHrmn, thank you for the detailed report. We will attempt to reproduce the issue and get back to you here with some guidance.

[JncHrmn]

@tyllark thanks for looking into this. I wanted to give you a quick status update on our internal testing efforts and share some additional context:

We shipped Ad-Hoc build to real iPhones. DataStore sync now behaves as expected. We see the full initial sync complete, queries return the correct data after the splash/loading screen, and changes made offline are properly pushed once connectivity returns. We still see intermittent timing/race issues under the simulator, but as long as it works in "production", this is fine I guess.

Unfortunately, on Android physical devices DataStore appears to be entirely non-functional. During/After the login the sequence of DataStore.clear() and DataStore.start() is firing and it seems that it is successful as well. At this point CRUD operations on the models are possible. But after closing and cold-restarting the app, all models return empty lists—no data is ever pulled down or pushed up and any new entries created in the app are written locally but never synchronized to the cloud. When re-opening the app again, the created data is lost. The only option to get back into a functional state is only by logging out and logging in again.

I hope this helps and thanks again for the support!

[tyllark]

Hello @JncHrmn, DataStore does not sync data on startup, so you would need to call a function such as Amplify.DataStore.start() to trigger a sync and the DataStore ready event on startup. From our docs:

Each time DataStore starts (via start or any other operation: query, save, delete, or observe), DataStore will reevaluate the syncExpressions.

As for your data not syncing could you please update your DataStore Hub events to include these logging statements to help us understand the status of the DataStore Outbox.

Amplify.Hub.listen(HubChannel.DataStore, (hubEvent) {
      String hubPayload = 'No HubPayload';
      switch (hubEvent.type) {
        case DataStoreHubEventType.ready:
          break; //No Payload
        case DataStoreHubEventType.networkStatus:
          final payload = hubEvent.payload as NetworkStatusEvent?;
          hubPayload = 'active: ${payload?.active}';
        case DataStoreHubEventType.subscriptionsEstablished:
          break; //No Payload
        case DataStoreHubEventType.syncQueriesStarted:
          final payload = hubEvent.payload as SyncQueriesStartedEvent?;
          hubPayload = 'models: ${payload?.models}';
        case DataStoreHubEventType.modelSynced:
          final payload = hubEvent.payload as ModelSyncedEvent?;
          hubPayload =
              'modelName: ${payload?.modelName}, isFullSync: ${payload?.isFullSync}, isDeltaSync: ${payload?.isDeltaSync}, added: ${payload?.added}, updated: ${payload?.updated}, deleted: ${payload?.deleted}';
        case DataStoreHubEventType.syncQueriesReady:
          break; //No Payload
        case DataStoreHubEventType.outboxMutationEnqueued:
          final payload = hubEvent.payload as OutboxMutationEvent?;
          hubPayload =
              'modelName: ${payload?.modelName}, elementType: ${payload?.element.model.runtimeType}';
        case DataStoreHubEventType.outboxMutationProcessed:
          final payload = hubEvent.payload as OutboxMutationEvent?;
          hubPayload =
              'modelName: ${payload?.modelName}, elementType: ${payload?.element.model.runtimeType}';
        case DataStoreHubEventType.outboxMutationFailed:
          break; //No Payload
        case DataStoreHubEventType.outboxStatus:
          final payload = hubEvent.payload as OutboxStatusEvent?;
          hubPayload = 'isEmpty: ${payload?.isEmpty}';
        case DataStoreHubEventType.subscriptionDataProcessed:
          final payload = hubEvent.payload as SubscriptionDataProcessedEvent?;
          hubPayload =
              'modelName: ${payload?.modelName}, elementType: ${payload?.element.model.runtimeType}, version: ${payload?.element.version}, lastChangedAt: ${payload?.element.deleted}, elementType: ${payload?.element.deleted}';
      }

      safePrint('DataStoreEvent - ${hubEvent.eventName} - $hubPayload');
    });

On my side I'm able to Update my DynamoDB table and sync without issue on my Physical Android device:

networkStatus - active: true
outboxStatus - isEmpty: true
outboxMutationEnqueued - modelName: Todo, elementType: Todo
outboxStatus - isEmpty: false
subscriptionsEstablished - No HubPayload
syncQueriesStarted - models: [Todo]
modelSynced - modelName: Todo, isFullSync: false, isDeltaSync: true, added: 0, updated: 0, deleted: 0
syncQueriesReady - No HubPayload
ready - No HubPayload
outboxStatus - isEmpty: false
subscriptionDataProcessed - modelName: Todo, elementType: Todo, version: 1, lastChangedAt: false, elementType: false
outboxMutationProcessed - modelName: Todo, elementType: Todo
3x outboxStatus - isEmpty: true
outboxStatus - isEmpty: true
subscriptionsEstablished - No HubPayload
syncQueriesStarted - models: [Todo]
104x subscriptionDataProcessed - modelName: Todo, elementType: Todo, version: 1, lastChangedAt: false, elementType: false
syncQueriesReady - No HubPayload
modelSynced - modelName: Todo, isFullSync: true, isDeltaSync: false, added: 104, updated: 0, deleted: 0
ready - No HubPayload
outboxStatus - isEmpty: true

[JncHrmn]

@tyllark Sorry for the late reply. So we did more testing with the android setup. We started testing only on the simulator and added the print statements you provided. There was nothing unusual and everything worked as intended. Then we started simulating on a real device but still connect to the PC via cable. This also worked as intended, especially with your comment in mind, on how the datastore starts the sync process. We even build a debug apk file and distributed them to a few team members. The app was noticeable slower (flutter related), but the datastore seemed to work. Local data was there after app starts and syncing with the cloud did also work. Logging in and out always loaded the data of the user.

Now we uploaded the app to the app store for beta testing and release. In both cases, the app is not usable at all. All the problems that were described before are happening there. User logs in for the first time. Datastore is running (models synced etc.) and the user can create entries, update or delete them. But the moment the app is closed and reopened, nothing seems to work. If a user creates an other entry and we would fully close the app and open it again. All the data from before is gone. Meaning query data or creating is not syncing or refetching is always providing empty lists, even though there is data for this given user in the DB. It feels strange, that the moment the app "enters" the play store, something with the datastore seems to break. We couldn't provide any print outs so far, but if this is necessary for you to further proceed, we try to add a solution.

[ekjotmultani]

This is strange, @JncHrmn are you using any publishing/ distribution service to the Play Store?

[NiklasMeetyu]

@ekjotmultani @tyllark Indeed, our app on android is working fine only on the Simulator and when running "flutter run" on a real device meaning running the debug apk. As soon as we download from the PlayStore the Datastore/App only works the first time the users opens (registers or logs in, CRUD operations). The n-th (n>1) time the app is opened, it breaks (no data is loading - neither locally nor from the cloud).

Here is the upload procedure for PlayStore release which we perform on a Mac M3 in VS Code.

1. We choose our correct cloud environments (e.g. amplify production env).
2. We choose our correct git prod code branch.
3. We follow the app signing as outlined in the documentation and nicely described in this video from "HeyFlutter" https://www.youtube.com/watch?v=g0GNuoCOtaQ This means we’re installing the keystore on our mac and add the necessary keystore files and code additions to the repo.
4. We run the terminal command: flutter build appbundle
5. We upload this appbundle in the Google Play Console and submit for review.
6. We download the app from the Store and run into the described issue.

Interestingly, we created also a release apk ourselves and installed on a real device. Observation: We were able to create an entry but then the app crashed right away (data was saved tho - as we saw it on app restart). From now on the app would work but only for around 3 - 10 seconds after opening it before shutting down even when just displaying the data.

[tyllark]

Hello @JncHrmn @NiklasMeetyu, we will need a minimal reproducible example or at the very least error logs to properly address this issue. Could you please try/answer the following to help us identify the source of the error.

1. Temporarily update your safePrint calls to print (please change this back before releasing to production) so we can get logs in release mode.
2. If you are not already can you please print errors received via FlutterError.onError and PlatformDispatcher.instance.onError to help identify the cause of the crash.
3. Are you able to reproduce the error by running flutter run --release or only from PlayStore/Side Loading APKs? flutter logs can be used to see print statements from side loaded APKs.
4. Does this issue only happen when pointing to the prod environment or can it be reproduced on your dev environment?

[JncHrmn]

Hi @tyllark. So we tried to do the steps you recommended. Regarding 4. This can be reproduced in dev as well as in prod env. But only on android and not on ios.

We tried flutter run --release and the app crashed at different points throughout the testing process. Here is the log after the app start:

D/FlutterLocationService( 2983): Binding to location service.
D/FlutterGeolocator( 2983): Flutter engine connected. Connected engine count 1
I/flutter ( 2983): DataStoreEvent - networkStatus - active: true
I/flutter ( 2983): DataStoreEvent - outboxStatus - isEmpty: false
I/flutter ( 2983): DataStoreEvent - subscriptionsEstablished - No HubPayload
I/flutter ( 2983): DataStoreEvent - syncQueriesStarted - models: [Chat, ChatSession, Message, JournalEntry, Calendar, Feature, ToDoItem, Label, ToDoItemLabel, Plan, UserPlan, PublicAIAssistants, Company, FirstEntryCreation, Mood, AppVersionControl, Topic, MoodRecording, UserScore, JournalEntryLabel, RankSystem, CustomAIAssistants, Multimedia, JournalEntryMultimedia, UserDeletionLog, TopicContentConnection, JournalPrompt, Quota, CalendarEvent, ToDoItemMultimedia, UserSettings, SystemStatus, UserContext, CompanyUserHistory, QuotaUsage]
I/flutter ( 2983): DataStoreEvent - modelSynced - modelName: Chat, isFullSync: false, isDeltaSync: true, added: 0, updated: 0, deleted: 0
... (All the models getting synced)
I/flutter ( 2983): DataStoreEvent - modelSynced - modelName: QuotaUsage, isFullSync: false, isDeltaSync: true, added: 0, updated: 0, deleted: 0
I/flutter ( 2983): DataStoreEvent - syncQueriesReady - No HubPayload
I/flutter ( 2983): DataStoreEvent - ready - No HubPayload
I/flutter ( 2983): DataStoreEvent - outboxStatus - isEmpty: false

After I/flutter ( 2983): DataStoreEvent - outboxStatus - isEmpty: false the app crashes. There was not always an error message, but when we got one, then this was printed to the console:

...
I/flutter (11074): DataStoreEvent - syncQueriesReady - No HubPayload
I/flutter (11074): DataStoreEvent - outboxStatus - isEmpty: false
E/AndroidRuntime(11074): FATAL EXCEPTION: A8.H Dispatcher
E/AndroidRuntime(11074): Process: app.meetyu.meetyuapp.dev, PID: 11074
E/AndroidRuntime(11074): java.lang.IllegalAccessError: Illegal class access: 'j1.i' attempting to access 'com.amplifyframework.datastore.syncengine.TimeBasedUuid' (declaration of 'j1.i' appears in /data/app/~~9qORSi_-b9NfX7OWQCS6nA==/app.meetyu.meetyuapp.dev-x5F9hIqyACoLQ269bDEmKA==/base.apk)
E/AndroidRuntime(11074):        at j1.i.w(SourceFile:19)
E/AndroidRuntime(11074):        at u7.a.h(SourceFile:49)
E/AndroidRuntime(11074):        at m7.a.g(SourceFile:6)
E/AndroidRuntime(11074):        at u7.c.h(SourceFile:100)
E/AndroidRuntime(11074):        at m7.a.g(SourceFile:6)
E/AndroidRuntime(11074):        at u7.i.h(SourceFile:10)
E/AndroidRuntime(11074):        at m7.a.g(SourceFile:6)
E/AndroidRuntime(11074):        at u7.i.h(SourceFile:10)
E/AndroidRuntime(11074):        at m7.a.g(SourceFile:6)
E/AndroidRuntime(11074):        at u7.i.h(SourceFile:10)
E/AndroidRuntime(11074):        at m7.a.g(SourceFile:6)
E/AndroidRuntime(11074):        at u7.j.onError(SourceFile:29)
E/AndroidRuntime(11074):        at u7.h.onError(SourceFile:40)
E/AndroidRuntime(11074):        at u4.c.onError(SourceFile:34)
E/AndroidRuntime(11074):        at w7.b.onError(SourceFile:8)
E/AndroidRuntime(11074):        at u4.c.onError(SourceFile:10)
E/AndroidRuntime(11074):        at y7.o.onError(SourceFile:14)
E/AndroidRuntime(11074):        at t7.g.onError(SourceFile:11)
E/AndroidRuntime(11074):        at E7.c.f(SourceFile:15)
E/AndroidRuntime(11074):        at y7.E.onError(SourceFile:24)
E/AndroidRuntime(11074):        at E7.c.f(SourceFile:15)
E/AndroidRuntime(11074):        at y7.u.c(SourceFile:28)
E/AndroidRuntime(11074):        at y7.u.g(SourceFile:4)
E/AndroidRuntime(11074):        at y7.u.f(SourceFile:7)
E/AndroidRuntime(11074):        at y7.t.onError(SourceFile:25)
E/AndroidRuntime(11074):        at q7.b.error(SourceFile:3)
E/AndroidRuntime(11074):        at y7.h.g(SourceFile:206)
E/AndroidRuntime(11074):        at m7.k.f(SourceFile:6)
E/AndroidRuntime(11074):        at y7.u.i(SourceFile:173)
E/AndroidRuntime(11074):        at y7.u.d(SourceFile:50)
E/AndroidRuntime(11074):        at J7.d.d(SourceFile:28)
E/AndroidRuntime(11074):        at J7.e.d(SourceFile:47)
E/AndroidRuntime(11074):        at y7.F.onError(SourceFile:12)
E/AndroidRuntime(11074):        at y7.l.onError(SourceFile:34)
E/AndroidRuntime(11074):        at t7.g.onError(SourceFile:11)
E/AndroidRuntime(11074):        at z7.e.onError(SourceFile:19)
E/AndroidRuntime(11074):        at u4.c.onError(SourceFile:18)
E/AndroidRuntime(11074):        at q7.b.error(SourceFile:4)
E/AndroidRuntime(11074):        at z7.b.c(SourceFile:42)
E/AndroidRuntime(11074):        at m7.s.b(SourceFile:1)
E/AndroidRuntime(11074):        at t7.f.c(SourceFile:35)
E/AndroidRuntime(11074):        at z7.a.c(SourceFile:33)
E/AndroidRuntime(11074):        at com.amplifyframework.datastore.syncengine.i.accept(SourceFile:1)
E/AndroidRuntime(11074):        at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$mutation$3(Unknown Source:16)
E/AndroidRuntime(11074):        at com.amplifyframework.datastore.appsync.AppSyncClient.a(SourceFile:1)
E/AndroidRuntime(11074):        at com.amplifyframework.auth.b.accept(SourceFile:1)
E/AndroidRuntime(11074):        at com.amplifyframework.api.aws.AppSyncGraphQLOperation$OkHttpCallback.onResponse(SourceFile:81)
E/AndroidRuntime(11074):        at F8.m.run(SourceFile:55)
E/AndroidRuntime(11074):        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
E/AndroidRuntime(11074):        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
E/AndroidRuntime(11074):        at java.lang.Thread.run(Thread.java:1012)

We tried the --release on multiple devices and on one of them we got this error during the datastore syncing, when a user logged in. This has not caused the app to crash, but stopped the syncing process which resulted in some models not being synced/created and therefore creating or reading data for these models did not work:

W/amplify:aws-datastore(12141): API sync failed - transitioning to LOCAL_ONLY.
W/amplify:aws-datastore(12141): GraphQLResponseException{message=Subscription error for QuotaUsage: [GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'AWSTimestamp' within parent 'QuotaUsage' (/onUpdateQuotaUsage/_lastChangedAt)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='_lastChangedAt'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'Int' within parent 'QuotaUsage' (/onUpdateQuotaUsage/_version)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='_version'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'Float' within parent 'QuotaUsage' (/onUpdateQuotaUsage/amountConsumed)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='amountConsumed'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'AWSDateTime' within parent 'QuotaUsage' (/onUpdateQuotaUsage/createdAt)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='createdAt'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'AWSDateTime' within parent 'QuotaUsage' (/onUpdateQuotaUsage/lastUpdated)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='lastUpdated'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'AWSDateTime' within parent 'QuotaUsage' (/onUpdateQuotaUsage/updatedAt)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='updatedAt'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'ID' within parent 'QuotaUsage' (/onUpdateQuotaUsage/userID)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='userID'}]', extensions='null'}], errors=[GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'AWSTimestamp' within parent 'QuotaUsage' (/onUpdateQuotaUsage/_lastChangedAt)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='_lastChangedAt'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'Int' within parent 'QuotaUsage' (/onUpdateQuotaUsage/_version)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='_version'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'Float' within parent 'QuotaUsage' (/onUpdateQuotaUsage/amountConsumed)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='amountConsumed'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'AWSDateTime' within parent 'QuotaUsage' (/onUpdateQuotaUsage/createdAt)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='createdAt'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'AWSDateTime' within parent 'QuotaUsage' (/onUpdateQuotaUsage/lastUpdated)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='lastUpdated'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'AWSDateTime' within parent 'QuotaUsage' (/onUpdateQuotaUsage/updatedAt)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='updatedAt'}]', extensions='null'}, GraphQLResponse.Error{message='Cannot return null for non-nullable type: 'ID' within parent 'QuotaUsage' (/onUpdateQuotaUsage/userID)', locations='null', path='[GraphQLPathSegment{value='onUpdateQuotaUsage'}, GraphQLPathSegment{value='userID'}]', extensions='null'}]
W/amplify:aws-datastore(12141): , recoverySuggestion=See attached list of GraphQLResponse.Error objects.}
W/amplify:aws-datastore(12141):         at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$subscription$1(AppSyncClient.java:315)
W/amplify:aws-datastore(12141):         at com.amplifyframework.datastore.appsync.AppSyncClient$$ExternalSyntheticLambda0.accept(D8$$SyntheticClass:0)
W/amplify:aws-datastore(12141):         at com.amplifyframework.api.aws.SubscriptionEndpoint$Subscription.dispatchNextMessage(SubscriptionEndpoint.java:461)
W/amplify:aws-datastore(12141):         at com.amplifyframework.api.aws.SubscriptionEndpoint.notifySubscriptionData(SubscriptionEndpoint.java:257)
W/amplify:aws-datastore(12141):         at com.amplifyframework.api.aws.SubscriptionEndpoint.-$$Nest$mnotifySubscriptionData(Unknown Source:0)
W/amplify:aws-datastore(12141):         at com.amplifyframework.api.aws.SubscriptionEndpoint$AmplifyWebSocketListener.processJsonMessage(SubscriptionEndpoint.java:647)
W/amplify:aws-datastore(12141):         at com.amplifyframework.api.aws.SubscriptionEndpoint$AmplifyWebSocketListener.onMessage(SubscriptionEndpoint.java:544)
W/amplify:aws-datastore(12141):         at [okhttp3.internal.ws](http://okhttp3.internal.ws/).RealWebSocket.onReadMessage(RealWebSocket.kt:392)
W/amplify:aws-datastore(12141):         at [okhttp3.internal.ws](http://okhttp3.internal.ws/).WebSocketReader.readMessageFrame(WebSocketReader.kt:255)
W/amplify:aws-datastore(12141):         at [okhttp3.internal.ws](http://okhttp3.internal.ws/).WebSocketReader.processNextFrame(WebSocketReader.kt:111)
W/amplify:aws-datastore(12141):         at [okhttp3.internal.ws](http://okhttp3.internal.ws/).RealWebSocket.loopReader(RealWebSocket.kt:307)
W/amplify:aws-datastore(12141):         at [okhttp3.internal.ws](http://okhttp3.internal.ws/).RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
W/amplify:aws-datastore(12141):         at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:529)
W/amplify:aws-datastore(12141):         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
W/amplify:aws-datastore(12141):         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:651)
W/amplify:aws-datastore(12141):         at java.lang.Thread.run(Thread.java:1119)
I/amplify:aws-datastore(12141): Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLY
I/amplify:aws-datastore(12141): Stopping subscription processor.
I/amplify:aws-datastore(12141): Stopped subscription processor.
I/amplify:aws-datastore(12141): Setting currentState to LOCAL_ONLY
I/amplify:aws-api(12141): No more active subscriptions. Closing web socket.

We are not sure if this might be related to the errors that happened before. But this QuotaUsage model is differently setup compared to our other models. User has just read access, but is never reading any data. We basically used it as a workaround.

Please let us know, if you need more informations

[tyllark]

Hello @JncHrmn, sorry for the delayed response but we are still investigating. I'm reaching out to the Amplify Android team to help investigatejava.lang.IllegalAccessError as they maintain the Android DataStore SDK. The Cannot return null for non-nullable type on QuotaUsage is interesting since it implies the data is in an invalid state. Can you please provide the following to help us continue investigating:

1. Your full GraphQL schema file.
2. What modifications have you made to your QuotaUsage schema? a. Did you change the types from nullable to non-nullable? b. Did you update any auth permissions? c. Did you run amplify push and regenerate your models amplify codegen models after making these updates?

[NiklasMeetyu]

Hello @tyllark Thank you for your follow-up. Here the answers:

1. We cannot disclose the full schema publicly here but would be willing using another way you offer. However, this what the affected type QuotaUsage looks like (please note, we need this type for our backend to handle access permissions to features the app offers. A normal user being a client on the app should not have access to any of this data. However, since not specifying a rule for Cognito users breaks the entire datastore on android (not iOS), we added the read owner-based permission which effectively gives a way for Cognito users to authenticate themselves with the graphql api but get no data back):

type QuotaUsage
  @model
  @auth(rules: [{ allow: private, operations: [read] },{ allow: private, provider: iam }]) {
  id: ID! @primaryKey
  userID: ID! @index(name: "byUser")
  featureName: String! @index(name: "byFeature")
  amountConsumed: Float!
  lastUpdated: AWSDateTime!
  feature: Feature @belongsTo(fields: ["featureName"])
}

2.a. No, we didn't change from nullable to non-nullable on this type. 2.b. No, we didn't update auth permission on this type. Throughout development auth rules were surely changed on other types, however. 2.c. Yes, we did run amplify push and also always do the amplify codegen models command

If that helps, we're happy to schedule a call to give you access to everything you need.

Thanks, Niklas and @JncHrmn

[tyllark]

Hello @NiklasMeetyu, thank you for clarifying. Based on your description I believe the issue may be related to how your models are interconnected. Could you please send your schema to my Amazon Discord tyllark_58391 and I will attempt to reproduce the issue with it.

[NiklasMeetyu]

Hi @tyllark We hope you're doing well! The issue in this thread is blocking android releases for over two months now and we have no resolution path in sight. This is a big blocker for our business development. We really hope we won't have to leave the amplify flutter ecosystem due to this.

Can you please share the current status and let us know when the datastore will be fixed to not break in production when distributed via Google Play?

Thanks a lot, and please let us know what else we can provide to help improve the software. Niklas

[JncHrmn]

As already shared but here is another error log from our analytics which was created upon the different app crashes. The iOS App is released with the exact same setup and he have not experienced any crashes that were related to the "appsync-engine".

Errorlog:

          Fatal Exception: java.lang.IllegalAccessError: Illegal class access: 'P2.b' attempting to access 'com.amplifyframework.datastore.syncengine.TimeBasedUuid' (declaration of 'P2.b' appears in /data/app/~~VmpUTHUY4KEs7uytDh_Svw==/(appIdentifier)==/base.apk)
       at P2.b.D(SourceFile:19)
       at j8.a.h(SourceFile:49)
       at b8.a.g(SourceFile:6)
       at j8.c.h(SourceFile:100)
       at b8.a.g(SourceFile:6)
       at j8.i.h(SourceFile:10)
       at b8.a.g(SourceFile:6)
       at j8.i.h(SourceFile:10)
       at b8.a.g(SourceFile:6)
       at j8.i.h(SourceFile:10)
       at b8.a.g(SourceFile:6)
       at j8.j.onError(SourceFile:29)
       at j8.h.onError(SourceFile:40)
       at com.google.android.gms.internal.measurement.w4.onError(SourceFile:18)
       at l8.b.onError(SourceFile:8)
       at n5.n.onError(SourceFile:10)
       at n8.o.onError(SourceFile:14)
       at i8.g.onError(SourceFile:11)
       at t8.c.f(SourceFile:15)
       at n8.E.onError(SourceFile:24)
       at t8.c.f(SourceFile:15)
       at n8.u.c(SourceFile:28)
       at n8.u.g(SourceFile:4)
       at n8.u.f(SourceFile:7)
       at n8.t.onError(SourceFile:25)
       at f8.b.error(SourceFile:3)
       at n8.h.g(SourceFile:206)
       at b8.k.f(SourceFile:6)
       at n8.u.i(SourceFile:173)
       at n8.u.d(SourceFile:50)
       at y8.d.d(SourceFile:28)
       at y8.e.d(SourceFile:48)
       at n8.F.onError(SourceFile:12)
       at n8.l.onError(SourceFile:34)
       at i8.g.onError(SourceFile:11)
       at o8.e.onError(SourceFile:19)
       at n5.n.onError(SourceFile:18)
       at f8.b.error(SourceFile:4)
       at o8.b.c(SourceFile:42)
       at b8.s.b(SourceFile:1)
       at i8.f.c(SourceFile:34)
       at o8.a.c(SourceFile:33)
       at com.amplifyframework.datastore.syncengine.i.accept(SourceFile:1)
       at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$mutation$3(:16)
       at com.amplifyframework.datastore.appsync.AppSyncClient.a(SourceFile:1)
       at com.amplifyframework.auth.b.accept(SourceFile:1)
       at com.amplifyframework.api.aws.AppSyncGraphQLOperation$OkHttpCallback.onResponse(SourceFile:81)
       at w9.j.run(SourceFile:55)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:651)
       at java.lang.Thread.run(Thread.java:1119)

[tyllark]

Hello @NiklasMeetyu and @JncHrmn, I apologize for losing track of this issue. Since there is a difference in behavior between the Android release and debug builds, code shrinking/obfuscation/optimization might be breaking something. Do you have ProGuard enabled in your Android project? If so can you update your /android/app/proguard-rules.pro to keep Amplify Datastore logic untouched in the release build:

-keep class com.amplifyframework.datastore.** { *; }
-keep class com.amplifyframework.datastore.syncengine.TimeBasedUuid { *; }

I have created an AWS backend with your schema and I'm working to reproduce the issue locally.

[JncHrmn]

@tyllark Thanks for the response. We currently have not enabled ProGuard. We will give this a try to see if it solves the issue.

[JncHrmn]

Hi @tyllark, we have set up ProGuard and pushed that into production. It seemed to work at first, because the previous crashes did not occur anymore, but now the whole DataStore is not functional anymore. For every model the user tries to query or create data we get the log Exception: Failed to start DataStore. We have not changed our setup, just added ProGuard.

[tyllark]

Hello @JncHrmn, I'm glad the crashing issue has been resolve, but I'm sorry the other DataStore issues still persist. I've been unable to reproduce the issue on my side thus far. I will provide a pubspec override in a couple hours that has verbose logging enabled for DataStore.

[tyllark]

Hi @JncHrmn, I created a branch that enables verbose logging for Amplify Android here. You can temporarily import the changes by updating your pubspec.yaml or pubspec_overrides.yaml to contain the following:

dependency_overrides:

  amplify_datastore:
    git:
      url: https://github.com/aws-amplify/amplify-flutter.git
      ref: test/datastore_logging
      path: packages/amplify_datastore

Please collect and provide the verbose logs using one of the following commands and I will work with the Amplify Android team to determine what is causing datastore to not start. We do not recommend deploying verbose logging builds to production.

adb logcat | grep -E "amplify:aws-datastore" > datastore_logs.txt

adb logcat | grep -E "amplify:aws-datastore|your_log_tag" > datastore_logs.txt

[NiklasMeetyu]

Hi @tyllark!

Thank you for creating the datastore verbose enabled version. We submitted a dedicated app release in closed beta to the Google Play Reviewer Team. Once approved, we'll create the log files and share them with you via Discord. Please let us know if you would like to put your or a team member's email in the closed beta access list.

Thank you, Niklas

[JncHrmn]

Hi @tyllark, we tested the datastore version you provided. As Niklas mentioned, we created a beta release to verify this version within our team and, in particular, on the devices that were previously affected. After three days of testing, none of us encountered the described issue anymore (the bug usually appeared after just one day). The app behaved as expected, and we didn’t see any datastore-related crashes. Therefore, at this point, we can’t provide any logs. Do you know if this datastore version differs from the one currently in production? Or could it be that a beta release on the Google Play Store is handled differently compared to a “real” production release? Otherwise, we might temporarily push the provided version into our production branch to see if the issue reappears there and to be able to capture the corresponding logs.

[harsh62]

@JncHrmn I discussed with @tyllark offline and we think you should not push to production with verbose logging enabled. There could be 2 things that might have happened:

• Some Android cache got cleared up and the issue is not occurring again.
• Verbose logging is somehow preventing the original race condition to NOT occur.

Next steps I would suggest to just use the latest version and test the issue again, making sure all the cache is cleared. If the issue is still happening, we would then have to investigate this further.