Way to get progress of sync

[ashiemke]

We need to load data to show the user when they land on our "home" page, so we need to wait for the datastore to sync before we can show the home screen. Loading all the objects (~1300) takes around 30s, which is a long time to look at a spinner. (https://github.com/aws-amplify/amplify-ios/issues/1155 would also help this go faster)

I'd love to have a determinant progress indicator, but I don't see any way to know how many objects each model sync is going to pull down until they complete.

On a side note - why's it so slow? It's a few MB of data, mostly just some ~20 dimensional point data used to generate plots and derive data, I didn't expect it to take so long to download.

[diegocstn]

hi @ashiemke, the current implementation doesn't provide an estimate of how many objects per model the DataStore engine is about to sync but perhaps you can leverage on the dispatched hub events to present a progress indicator to your users:

https://docs.amplify.aws/lib/datastore/datastore-events/q/platform/ios

The syncQueriesStarted event contains the array of model names and for each model a modelSynced event will be dispatched.

[ashiemke]

yeah, I'm using that right now, but some models have 1 instance and some (one in particular) have thousands, so the time to sink each table is very different.

[ruiguoamz]

Hi @ashiemke.

My understanding of your use case is: For some model with large number of data, you don't want your users to keep looking at the spinner or empty view until the model finish syncing.

I have a workaround to faster display the synced model which is unrelated to the progress

// This is the first query that trigger the start of Sync Engine, and also see if there are data already exist locally
DataStore.query(Model.self) {
    // Added the newly queried data and re-render your view 
}
// Set up a real time listener for a specific model, so that if there are new data comes in that does not exist locally, the receiveValue block is executed
DataStore
    .publisher(for: Model.self) // https://docs.amplify.aws/lib/datastore/real-time/q/platform/ios#observe-data-in-real-time
    // `throttle`/`collect` here is to reduce the frequency of sink callback. For example, 1000 items goes through the `DataStore.publisher`, with throttle or collect, sink is trigger 100 times to avoid excessive `DataStore.query`
    .throttle(for: 10.0, scheduler: RunLoop.main, latest: true) // or
    .collect(.byTimeOrCount(DispatchQueue.main, time in seconds, count in integer))
    .sink {
        // Do something here
    }
    receiveValue: {
        DataStore.query(Moel.self) { 
            // Added the newly queried data and re-render your view however you want.
        }
    }

Hope this helps you, if not, feel free to provide more details. For now, I see this issue is a feature request.

[ashiemke]

@ruiguoamz Yeah, I have some code like that, but I'd need to control the order to make that work. Specifically I need to have the user model loaded before I'm able to display anything, and since the order can't be defined (https://github.com/aws-amplify/amplify-ios/issues/1155), it seems like I have to wait for a bunch of stuff to load.

[anatomybook]

yeah, I'm using that right now, but some models have 1 instance and some (one in particular) have thousands, so the time to sink each table is very different.

One possible workaround that I've used for this issue is to store the number of items from the last run. For my use-case the number of items between syncs doesn't change dramatically. Hence, it is usable to show a progress bar based on the total number of items from the previous run.

[ruiguoamz]

The thing I can think of for this feature is: DynamoDB is the source of truth of your data, every time local needs to sync with cloud, DataStore Sync Engine needs the number of entries of each model. From what I know so far, this total count is not returned from the backend. Feels like this is not only a SDK feature request, but also requires backend support.

I don't have a better solution so far and yes the order can't be defined because DataStore needs to ensure data consistency. Perhaps you can try what @anatomybook suggested.

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity. Please, provide an update or it will be automatically closed in 7 days.