Allow client to perform delta sync manually

[airstance]

There are times when my application is notified that a batch operation has been performed on the appsync datasource independent of the graphQL API (notifications via subscriptions). There is no way to request an existing sync operation to make a full/delta sync in this case. The only catch-up events are app foreground and network transitions

Describe the solution you'd like Provide a public interface to AppSyncSubscriptionWithSync to request a sync operation

Describe alternatives you've considered I would rather not make a manual request as the next delta sync would have to process the same data on the next operation.

An alternative would be to allow the client to update the SyncStrategy variable to allow updates to the baseRefreshIntervalInSeconds as well as lastSyncTime. Although adding the ability to adjust the base time has merit, I think allowing direct access to lastSyncTime is a bit messy.

Additional context I've added the following code to AppSyncSubscriptionWithSync.swift. Depending on how often it's called, it would be cleaner with "enhanced subscription state tracking".

func syncNow() {
    self.internalStateSyncQueue.addOperation {
        AppSyncLog.debug("Client request, syncing")
        self.performSync()
    }
}

[rohandubal]

Hello @airstance

To make sure I understand the problem correctly, could you please respond to some questions:

• The batch operation which happened, why would it not be part of the subscription updates? or even the catch up(delta) query?
• If the batch requests are fetched through catch up query then the end state would have all the results in lesser bandwidth right?
• Is the API implemented as per delta sync recommendations?
• Is the subscription which you are mentioning part of DeltaSync or some other subscription?

Thanks, Rohan

[airstance]

1. The batch operation occurs as part of a periodic large ETL transaction with 1000s of updates per second. The process updates the DynamoDB tables (data & deltasync) directly. These batch changes are caught by the catch up query but not the subscriptions.

2. Yes, I'm using the catch up query to collect all the changes in a single call.

3. The API is implemented as per the recommendations for delta sync.

4. The subscription I'm mentioning in the request is separate from the catch up subscription.

The batch operation bypasses the GraphQL mutation API because that would send 1000s of subscription notifications across 1000s of clients. They would all then make fetch requests for the full objects (because subscriptions would not contain the full object). Instead, I send a single "notification mutation" that is sent to the appropriate clients. They then use syncNow() to request a catch up sync which fetches all the necessary fields. This is provides a HUGE increase in resource efficiency.

I suppose I can force all mutations to request the entire object in the response. That would avoid the extra fetch requests. Even then, 1000s of subscription notifications across 1000s of clients is not great. The syncNow() solution reduces that to a single notification per client with the benefit of allowing the existing delta sync code to process the results as normal.

[airstance]

Here's the code I added to AppSyncSubscriptionWithSync.swift:

public protocol DeltaSync: class {
    func syncNow()
}

final class AppSyncSubscriptionWithSync<Subscription: GraphQLSubscription, BaseQuery: GraphQLQuery, DeltaQuery: GraphQLQuery>: Cancellable, DeltaSync {
.
.
.
    func syncNow() {
        self.internalStateSyncQueue.addOperation {
            AppSyncLog.debug("Client request, syncing")
            self.performSync()
        }
    }
.
.
.
}

[palpatim]

@airstance

I'm not understanding why DeltaSync as currently written doesn't meet this use case. You could set up a sync subscription to the "notification mutation", with an appropriate base query and sync query similar to what you'd perform during your proposed syncNow(). You would maintain an "incremental" subscription for user-generated mutations and catchups (presumably what you already have), and a second "bulk" subscription for handling the notification of ETL job completion.

Am I missing something in your use case?

[airstance]

That's effectively what I'm using now. I have a subscription tied to the sync operation that catches the "standard" singleton mutations. I then use a second subscription to catch the "bulk" operations notification.

As you say, I could simply start a fetch of all the objects that have changed since the bulk notification or use a list of objects sent in the subscription body. The reasons I wanted to use the existing delta sync code are twofold:

1. The code I would have to write is identical to the delta sync code. I would have to keep a "last updated" flag in persistent storage and fetch and update appropriately. This wouldn't be terribly hard but seems like a waste.

2. The bigger issue is that the incremental "bulk" fetch may download 1000s of items. It seems very inefficient to download them during the bulk subscription handler and again during the delta sync handler.

That brought me to a philosophical conclusion that the decision of when to start a delta sync operation might not lay solely in the hands of the AppSync code (currently based on reachability transitions). The application may have information that would help determine when a delta sync operation should be triggered.

The catch I see is that the delta sync code currently assumes a network reachability transition has occurred and (very inefficiently) does some bookkeeping on MQTT sessions. That should probably be cleaned up anyway.

I toyed with the alternative of allowing the application to set the "last updated" time, but that's really bad because the application can, and probably did, miss some offline transactions and skipping ahead will lose them until the next complete resync.

[atierian]

Thank you for opening this issue. AWS AppSync SDK for iOS entered maintenance mode in September 2023 and will receive no further updates as of September 2024.

Please use Amplify Swift going forward. For information on upgrading to Amplify Swift, refer to the Upgrade from AppSync SDK documentation.