DataStore.stop or DataStore.clear may interrupt DataStore.start

[Rayv1]

Description

Hi,

iam trying to use datastore but cant get Selective Sync to work. I want to change my userId / sub to logged-in users id. Additionally if would add an updateAt Sort Key, which is not inside Code yet.

Here is my Code:

// Used to identify the owner, which is not present at start.
var sub = '';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatefulWidget {
  const MyApp({Key? key}) : super(key: key);

  @override
  _MyAppState createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
  bool isBusy = true;
  @override
  void initState() {
    _configureAmplify();
    super.initState();
  }

  Future<void> _configureAmplify() async {
    try {
      AmplifyDataStore _dataStorePlugin = AmplifyDataStore(
          modelProvider: ModelProvider.instance,
          syncExpressions: [
            DataStoreSyncExpression(FoodTransaction.classType, () {
              return FoodTransaction.OWNER.eq(sub);
            })
          ]);

      // add Amplify plugins
      await Amplify.addPlugins([_dataStorePlugin]);

      // configure Amplify
      //
      // note that Amplify cannot be configured more than once!
      if (!Amplify.isConfigured) {
        await Amplify.configure(amplifyconfig);
        setState(() {
          isBusy = false;
        });
      }
    } catch (e) {
      // error handling can be improved for sure!
      // but this will be sufficient for the purposes of this tutorial
      print('An error occurred while configuring Amplify: $e');
    }
  }

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Amplified Todo',
      home: isBusy ? CircularProgressIndicator() : TodosPage(),
      // builder: Authenticator.builder(),
    );
  }
}

class TodosPage extends StatefulWidget {
  @override
  _TodosPageState createState() => _TodosPageState();
}

class _TodosPageState extends State<TodosPage> {
  // loading ui state - initially set to a loading state
  bool _isLoading = true;

  // list of Todos - initially empty
  List<FoodTransaction> _todos = [];

  @override
  void initState() {

    super.initState();
  }

  @override
  void dispose() {
    // to be filled in a later step
    super.dispose();
  }



  changeSync() async {
    sub = 'facebook_xxxx';
    await Amplify.DataStore.stop();
    await Amplify.DataStore.start();
    _todos = await Amplify.DataStore.query(FoodTransaction.classType);
    setState(() {
      if (_isLoading) _isLoading = false;
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('My Todo List'),
      ),
      // body: Center(child: CircularProgressIndicator()),
      body: _isLoading
          ? Center(child: CircularProgressIndicator())
          : TodosList(todos: _todos),
      floatingActionButton: FloatingActionButton.extended(
        onPressed: () async {
          changeSync();
        },
        tooltip: 'change sync',
        label: Row(
          children: [Icon(Icons.add), Text('Add todo')],
        ),
      ),
      floatingActionButtonLocation: FloatingActionButtonLocation.centerFloat,
    );
  }
}

I Checked my Appsync Cloudwatch Logs and cant see any new sync or query action after my changeSync Method. I guess iam just missing some code or something else. I also noted that i cant use QueryPredicate.None because its not implemented as in Amplify-android, right?

Categories

• [ ] Analytics
• [ ] API (REST)
• [ ] API (GraphQL)
• [ ] Auth
• [ ] Authenticator
• [X] DataStore
• [ ] Storage

Steps to Reproduce

No response

Screenshots

No response

Platforms

• [ ] iOS
• [X] Android

Environment

[✓] Flutter (Channel stable, 2.10.3, on Ubuntu 20.04.4 LTS 5.13.0-37-generic, locale en_US.UTF-8)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
[✗] Chrome - develop for the web (Cannot find Chrome executable at google-chrome)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[✓] Android Studio (version 2021.1)
[✓] Android Studio
[✓] VS Code
[✓] Connected device (1 available)
[✓] HTTP Host Availability

! Doctor found issues in 1 category.

Dependencies

- amplify_api 0.4.1 [amplify_api_plugin_interface amplify_core collection flutter meta plugin_platform_interface]
- amplify_auth_cognito 0.4.1 [flutter amplify_auth_plugin_interface amplify_core collection plugin_platform_interface]
- amplify_authenticator 0.1.0 [amplify_auth_cognito amplify_auth_plugin_interface amplify_core amplify_flutter collection flutter flutter_localizations intl]
- amplify_core 0.4.1 [flutter plugin_platform_interface collection date_time_format meta uuid]
- amplify_datastore 0.4.1 [flutter amplify_datastore_plugin_interface amplify_core plugin_platform_interface meta collection async]
- amplify_flutter 0.4.1 [amplify_analytics_plugin_interface amplify_api_plugin_interface amplify_auth_plugin_interface amplify_core amplify_datastore_plugin_interface amplify_storage_plugin_interface collection flutter json_annotation meta plugin_platform_interface]
- cupertino_icons 1.0.4
- flutter 0.0.0 [characters collection material_color_utilities meta typed_data vector_math sky_engine]
- hive 2.1.0 [meta crypto]
- hive_flutter 1.1.0 [flutter hive path_provider path]
- path_provider 2.0.9 [flutter path_provider_android path_provider_ios path_provider_linux path_provider_macos path_provider_platform_interface path_provider_windows]

transitive dependencies:
- amplify_analytics_plugin_interface 0.4.1 [amplify_core flutter meta]
- amplify_api_plugin_interface 0.4.1 [amplify_core collection flutter json_annotation meta]
- amplify_auth_plugin_interface 0.4.1 [flutter meta amplify_core]
- amplify_datastore_plugin_interface 0.4.1 [flutter meta collection amplify_core]
- amplify_storage_plugin_interface 0.4.1 [flutter meta amplify_core]
- async 2.8.2 [collection meta]
- characters 1.2.0
- clock 1.1.0
- collection 1.15.0
- crypto 3.0.1 [collection typed_data]
- date_time_format 2.0.1
- ffi 1.1.2
- file 6.1.2 [meta path]
- flutter_localizations 0.0.0 [flutter intl characters clock collection material_color_utilities meta path typed_data vector_math]
- intl 0.17.0 [clock path]
- json_annotation 4.4.0 [meta]
- material_color_utilities 0.1.3
- meta 1.7.0
- path 1.8.0
- path_provider_android 2.0.12 [flutter path_provider_platform_interface]
- path_provider_ios 2.0.8 [flutter path_provider_platform_interface]
- path_provider_linux 2.1.5 [ffi flutter path path_provider_platform_interface xdg_directories]
- path_provider_macos 2.0.5 [flutter path_provider_platform_interface]
- path_provider_platform_interface 2.0.3 [flutter platform plugin_platform_interface]
- path_provider_windows 2.0.5 [ffi flutter path path_provider_platform_interface win32]
- platform 3.1.0
- plugin_platform_interface 2.1.2 [meta]
- process 4.2.4 [file path platform]
- sky_engine 0.0.99
- typed_data 1.3.0 [collection]
- uuid 3.0.6 [crypto]
- vector_math 2.1.1
- win32 2.4.4 [ffi]
- xdg_directories 0.2.0+1 [meta path process]

Device

Pixel 5 API 31

OS

Android 11

CLI Version

7.6.26

Additional Context

No response

[HuiSF]

Hi @Rayv1 thanks for reporting this issue.

Regarding your changeSync function implementation - You invoked query right after Amplify.DataStore.start(). At this moment, the data sync may be in still progress, where the query may return empty result as the data hasn't been written into local DB. To get around of this you have two options:

• Start querying when Amplify Hub emits DataStore ready event e.g.

  hubSubscription = Amplify.Hub.listen([HubChannel.DataStore], (msg) { 
    if (msg.eventName == "ready") { 
      _todos = await Amplify.DataStore.query(FoodTransaction.classType);
    } 
  }); 
  

• To use observeQuery API

In addition, for your use case, you may want to use Amplify.DataStore.clear() instead of Amplify.DataStore.stop() to clear previously synced data against the previous user sub

I Checked my Appsync Cloudwatch Logs and cant see any new sync or query action after my changeSync Method.

I will test and verify this point. Can you share your FoodTransaction model schema?

I also noted that i cant use QueryPredicate.None because its not implemented as in amplify-android, right?

QueryPredicate.None is not implemented in iOS precisely. Please do not use this constant as it's not officially documented.

[HuiSF]

Update re:

I Checked my Appsync Cloudwatch Logs and cant see any new sync or query action after my changeSync Method.

I created an App to replicate @Rayv1 provide steps. I noticed that consecutively invoking DataStore.clear() and DataStore.start(), may cause an issue.

• DataStore.clear() starts DataStore clearing process, this process is asynchronous
• DataStore.start() instructs DataStore to start API sync, and this process is asynchronous

It may happen that, when the sync engine starts subscription, the clear process has just completed, where it instructs terminate all ongoing subscription, therefore, the clear interrupts the start. I think amplify-flutter lacks a "locking" mechanism to prevent this from happening. I will look into a fix.

In the meantime, here's a workaround:

  changeSync() async {
    sub = 'facebook_xxxx';
    await Amplify.DataStore.stop();
    // add a manual delay to ensure the previous stop call not to interrupt the next start call
    await Future.delay(const Duration(seconds: 2));
    await Amplify.DataStore.start();
    _todos = await Amplify.DataStore.query(FoodTransaction.classType);
    setState(() {
      if (_isLoading) _isLoading = false;
    });
  }

[HuiSF]

Did a thorough testing in both Android and iOS platforms. Above mentioned issue is not reproducible in iOS. Looking at the source code of amplify-android of the stop API and clear API, it looks like that the onComplete action is not guaranteed to be executed after the stop or clear process is completely finished.

Therefore, in the amplify-flutter implementation, may not be effective.

Created an issue https://github.com/aws-amplify/amplify-android/issues/1690 for amplify-android for a further triaging.

[Rayv1]

HI @HuiSF

thanks for your research on my issue.

Did a thorough testing in both Android and iOS platforms. Above mentioned issue is not reproducible in iOS. Looking at the source code of amplify-android of the stop API and clear API, it looks like that the onComplete action is not guaranteed to be executed after the stop or clear process is completely finished.

Thats exactly what i saw on my Tests and thought its an issue on my site.

A brief feedback for Datastore: It seems like there is still much work to do on Amplify datastore and not useful for my app or larger apps. I plan with ~15000 Entries in Datastore , consisting of 15 different types / models and for me it seems that datastore is not ready for such data. I want to control each type of data separate to clear or start the sync expression for it and not for the all Data. Iam not sure what happens when i have 15 different sync expressions and restart the datastore plugins so much, because i want to add a new 'updated_at' timestamp to get only the newest data.

Iam also wondering why i cant do a new dynamic sync expression without to do a datastore restart. Additionally the async Datastore actions which cant be awaited makes the whole datastore thing max complex. Finally i ask myself why the default sync would do a full table scan, which is really the last thing you should do against dynamodb and increases costs. How long should the scan take when you handle more data and not just 100 items? As now and for me its easier to integrate the logic myself instead of using datastore.

Cheers Ray

[HuiSF]

Thanks for the write up @Rayv1 with details and your thoughts.

I want to control each type of data separate to clear

Unfortunenatly, this is not currently supported by DataStore, you are welcome to open a feature request for this. There is a workaround is that to use the delete API to delete the record that you want to clear.

I am also wondering why i cant do a new dynamic sync expression without to do a datastore restart.

The syncExpression is being re-evaluated every time when DataStore subscription event comes. Which will behave as a client filter to determine whether merge the subscription event into local DB.

If you want the updated syncExpression to be effective for the base/delta sync, you'd need to restart DataStore.

because i want to add a new 'updated_at' timestamp to get only the newest data

I may need more details on this use case... but you meant your App doesn't need data before the DataTime presented by updated_at? If this is true, you may try:

• Flip _deleted value to true in DynamoDB for the records that has updated_at < the desired value (this can be done in various ways, such a lambda function, custom API with AppSync etc)

• When App starts the base/delta sync will reconsolidate remote data (with _deleted = true) which will eliminate corresponding records from the local DB

• When App receives subscription event triggered by updating the _deleted field, DataStore reconciliation eliminate record from local DB with _deleted=true

• Specific updated_at as a Global Secondary Index (GSI) (document for configuring GSI) e.g.

  type FoodTransaction @model {
    id: ID!
    owner: String!
    updated_at: AWSDateTime @index(name: "byUpdatedAt", sortKeyFields: ["owner"])
  }
  

  final datastore = AmplifyDataStore(
    modelProvider: ModelProvider.instance,
    syncExpressions: [
        DataStoreSyncExpression(
          FoodTransaction.classType,
          () => FoodTransaction.UPDATED_AT.gt(TemporalDateTime(DateTime(2022, 3, 26))),
        ),
      ],
  )
  

Datastore actions which cant be awaited makes the whole datastore thing max complex.

I agree, sorry for the inconvenience, but to clarify, this unfortunately a bug is in amplify-android, we anticipate a fix soon.

• All DataStore operations are await-able in iOS platform
• Except stop and clear, other operations are await-able in Android platform

Finally i ask myself why the default sync would do a full table scan,

DataStore base/delta sync supports Query operation. To enable Query, you'd need to define GSI on your model schema, and use a syncExpression that matches the GSI (above code example). Details please refer to this document.

As now and for me it's easier to integrate the logic myself instead of using datastore

We'd love to hear your thoughts on how to improve DataStore, please feel free to open feature request with description of your use cases. :)

[Rayv1]

Hi @HuiSF

thanks for your reply. Can i await the Datastore to be full "synced"? Just for example..when i would store UserProfiles with DynamoDB and a user who already has a valid UserProfile would login with a new device with no local UserProfile, how can i await, that the UserProfile is loaded and synced, before showing the first screen? Are there events when a specific model is full synced?

[Jordan-Nelson]

@Rayv1 There are events emitted for each model type. See ModelSynced Event.

[harrynguyen2510]

I have this struggling issue for several months, that in the case one user logs out the app (1st user), another sign in app (2nd user), same device 1/ If I don't run Amplify.datastore.clear() after logout, the process of sign in with another account will be fast, because the sync hub already ready. But, second user can meet inaccurate local data from first user, as data leaking 2/ If I run Amplify.datastore.clear() after logout, it ensures that all data from first user are gone, avoiding data leaking issue, but it takes so long for sync hub get ready. In the case of iOS, taking around 15 seconds, while Android can be 2 minutes to sign in back. That's impractical for user

Currently I use method 1 to have good UX, but still high risk of data leaking. Is there any way that we can keep the sync hub always ready or any sync engine, but only clear data from schema models? Thank you

[HuiSF]

Hi @harrynguyen2510 DataStore currently doesn't have a selective clear functionality, here's the feature request issue: https://github.com/aws-amplify/amplify-flutter/issues/2528

[harrynguyen2510]

@HuiSF Thanks, hope the team can add more options on data clear features in the future

[wojciechzahradnikdeviniti]

Hi, any updates on this issue? I still have a problem with that.

[dgagnon]

I want to chime in with what we have tried, the issues we have faced and so far what worked and what does not. I thought this relates to this issue, but if you prefer I open a new one, let me know.

1. no await on close: can be solved this inelegant way (adjust timing as needed):

    Timer(const Duration(seconds: 1), () async {
      try {
        await Amplify.DataStore.stop();
        Timer(const Duration(seconds: 3), () async {
          try {
            await Amplify.DataStore.start();
          } on Exception catch (error, stackTrace) {
            err('Error starting DataStore: $error', stackTrace, cat: 'DATASTORE');
            if (kDebugMode) rethrow;
          }
        });
      } catch (error, stackTrace) {
        err('Error stopping DataStore: $error', stackTrace, cat: 'DATASTORE');
        if (kDebugMode) rethrow;
      }
    });

2. Be careful with syncExpressions that are referencing models with foreign keys, especially when mixed with MaxRecordSize. If not, you will get foreign key errors since some records are missing.

3. Moved to #3703

[wojciechzahradnikdeviniti]

Thanks - fixed in 1.4.0

[lucasbourne]

I'm still getting this issue on 1.4.0. I see this was fixed in Amplify Android 2.12.0, is this available in 1.4.0 or are we still waiting on these changes?

[Jordan-Nelson]

@LucasBourne Amplify Flutter v1.4.0 does include the fix in Amplify Android v2.12.0. However, as noted on the Amplify Android issue, there are limitations with the current fix.

Are you calling stop or clear more than once in a row? Could you share the code you use when stopping/starting DataStore?

[Jordan-Nelson]

As of v1.4.0, calling stop (or clear) and then start should not result in an issue. As noted in https://github.com/aws-amplify/amplify-android/issues/1690#issuecomment-1692358164 and https://github.com/aws-amplify/amplify-android/issues/1464, you may still see issues when calling clear/stop and then start multiple times in a row or when calling clear multiple times in a row. You can follow https://github.com/aws-amplify/amplify-android/issues/1464 for updates on that issue.

[stevegaunt]

@Jordan-Nelson what version of amplify flutter is this fixed in?? we still see this with amplify flutter 1.4.0 after doing an amplify datastore.clear(), we have had to put a 10 second delay before it can start again ( which is not ideal - especially on signing up as a different user)

we are using sync expressions , so initially on startup we gave to sync against user_id being 0, then when the user signs in we then stop/clear/start to the sync expression users the correct username token from auth to pull the changes from dynamodb.

[Jordan-Nelson]

@stevegaunt can you share the code you are using to clear and then restart DataStore?

[stevegaunt]

@Jordan-Nelson

Here's a snipped from the signing in function when the OTP is entered

Future submitOtp({ required String otp, required String requestValue, }) async {

//other logic //then the amplify stuff to stop/start to pick up sync expression change.. then clear if we have a new cognito id. if (authenticationStore.cognitoId != null) { final cogSubId = authenticationStore.cognitoId!.userSubResult.value; final hasChanged = AmplifyInitialisation.cognitoId != cogSubId; await Amplify.DataStore.stop(); //force resync. if (hasChanged) { if (Platform.isAndroid) { await Future.delayed(const Duration(seconds: 2)); } await Amplify.DataStore.clear(); if (Platform.isAndroid) { await Future.delayed(const Duration(seconds: 10)); } } await Amplify.DataStore.start(); } return signInResult.isSignedIn;