Mappings with multi process support

[maksadavid]

I am trying to read and write to the same database file from the app and an app extension. I enabled enableMultiProcessSupport in database options but I still have a crash when I call [YapDatabaseViewConnection getSectionChanges:rowChanges:forNotifications:withMappings:] inside the app while the app extension is modifying the database.

*** Terminating app due to uncaught exception 'YapDatabaseException', reason: 'ViewConnection[0x2808badf0, RegisteredName=DBConversationsViewExtensionName] was asked for changes, but given mismatched mappings & notifications.'

I saw that there is a YapDatabaseModifiedExternallyNotification that the app can listen to but I couldn't find any example of how it is supposed to work.

[maksadavid]

I partially solved my previous problem by preventing the crash in case an update is missing.

    NSArray *notifications = [self.databaseConnection beginLongLivedReadTransaction];

    if (notifications && notifications.count > 0) {
         NSDictionary *firstChangeset = [[notifications objectAtIndex:0] userInfo];
         uint64_t firstSnapshot = [[firstChangeset objectForKey:YapDatabaseSnapshotKey] unsignedLongLongValue];
         if (self.messageMappings.snapshotOfLastUpdate != firstSnapshot - 1) {
             [self.databaseConnection readWithBlock:^(YapDatabaseReadTransaction *transaction) {
                 [self.messageMappings updateWithTransaction:transaction];
             }];
             // reload table view data
             return;
         }
     }

I am still having problems that might be related.

I noticed that depending on the database connection I am using the content of the YapDatabaseAutoView is different. This happens when the object is saved inside the iOS Notification Service app extension and happens more frequently if the database view contains only a few objects. If I restart the app everything is back to normal.

[robbiehanson]

You've stumbled on several issues within YapDatabase that I've been wanting to fix for some time now.

The general problem is that YapDatabase caches a bunch of stuff in-memory. This makes it very FAST, but also requires some book-keeping to ensure everything is in sync.

Each YapDBConnection is independent from the others, and has its own set of caches. So connectionA & connectionB have completely separate caches. Now if connectionA & connectionB are in the same process, then YapDB automatically handles cross-connection updates for you. That is, if connectionA modifies an item in the database (e.g. {collection=foo, key=bar}), then a change-set is created that reflects this change. The change-set is automatically propagated to other connections within the same process. Thus when connectionB moves forward to the new commit, it can process the corresponding change-set, and automatically update its internal caches to match the new reality on disk. (The beauty of this is that connectionB doesn't need to flush its caches. It can flush just the changed items. Or even better: it can update its caches with the new version of the object.)

The difficulty arrises when connectionA & connectionB are in different processes. And there are 2 specific problems to solve:

1. How does connectionB determine that connectionA performed a read-write transaction?
2. How does connectionB go about updating its caches?

The way it works right now is rather primitive.

If you setup the CrossProcessNotification extension, then you can send a signal from processA to processB that says basically "hey, I made a change to the database". This is helpful, but limited.

It allows you (and the database) to react similarly to a YapDatabaseModified notification. However things don't run as smoothly as if the modification was in the local process. Specifically because the corresponding change-set is missing. And because of this:

• all connections will be forced to dump ALL in-memory caches
• and things like Views won't be able to provide detailed change-sets (which is the exact problem you encountered)

I have a half-implemented solution that should work better:

• if you enable multiProcessSupport
• then recent change-sets are written to the database during a commit
• this allows other processes to automatically receive the change-set skeleton
• which gives other connections nearly everything they need to update as if the change occurred within the local process

I noticed that depending on the database connection I am using the content of the YapDatabaseAutoView is different. This happens when the object is saved inside the iOS Notification Service app extension and happens more frequently if the database view contains only a few objects. If I restart the app everything is back to normal.

Is one of these database connections using a longLivedReadConnection? And the other one isn't?

If so it's likely the following is occurring:

• the database starts at commit 42
• connectionA is using a longLivedReadConnection, and is at commit 42
• a different process performs a read-write transaction, that moves the database's latest commit to 43
• connectionB performs read transaction, and sees the database at commit 43
• since there was NO CrossProcessNotification, connectionA is still sitting at commit 42

[ksuther]

We ran in to similar issues with multiprocess support as documented in #471. Rather than writing changesets to the database, we experimented with using XPC to pass changesets in-memory. This would allow for the caches to stay up to date across processes without doing more work.

In the end we ended up not going that route because having the other processes dump the caches and reload turned out to not be as big of a performance problem than we feared (on Mac). We didn't have any YapDatabase views to worry about either, which made things a bit easier.

I'm not adding much new information, just adding on to your comment to say that we've run into similar issues but are also successfully using multiprocess support on both Mac and iOS in a fairly data-intensive app (Fantastical).

[robbiehanson]

Rather than writing changesets to the database, we experimented with using XPC to pass changesets in-memory. This would allow for the caches to stay up to date across processes without doing more work.

We considered XPC too, but there were potential security issues that concerned us.

The current idea is to write a SKELETON of the change-set to disk. This doesn't include objects. It's really just a minimal change-set that includes mostly rowid's. And we have a max number of recent change-sets that will be kept on disk at any one time. For example, only the most recent 20. Or perhaps anything within the last few minutes. This keeps the disk requirements low. And, of course, we only do this if multiProcessSupport is enabled.

[maksadavid]

Thanks for the detailed response. I created a mini project to demonstrate the issues I am currently having. https://github.com/maksadavid/MultiProcessApp

All connections seems to be at the same commit (i.e. [YapDatabaseConnection snapshot] is the same) but their internal caches are out of sync.

I have made a quick fix by changing one line of code in YapDatabaseViewTransaction.m. Line 229 I set BOOL shortcut = NO; I am using YapDatabase/SQLCipher 3.1.4 with Cocoapods. This seems to have no major impact on performance but solves the issue of having databaseConnection caches out of sync. It does not solve the crash I am having with mappings. For this the only fix I found was to try catch the exception and reload the mappings completely. The notification app extension usually doesn't modifies the database while the app is running. This is ok for us for the moment but I would appreciate to have a better fix.

[aadilHaffez]

Hi @robbiehanson We ran into the same issue while implementing Notification extension. We have enabed the "enableMultiProcessSupport", all YapDatabaseView's are Non-persistent.

We added observer on "YapDatabaseModifiedExternallyNotification",

YapDatabaseCrossProcessNotification* cpn = [[YapDatabaseCrossProcessNotification alloc] initWithIdentifier:@"crossnotification"];
[self.database registerExtension:cpn withName:@"crossprocess"];

[[NSNotificationCenter defaultCenter] addObserverForName:YapDatabaseModifiedExternallyNotification object:self.database queue:[NSOperationQueue mainQueue] usingBlock:^(NSNotification * _Nonnull note) {
       NSArray <NSNotification *>*changes = [weakSelf.longLivedReadOnlyConnection beginLongLivedReadTransaction];
               if (changes != nil) {
                   [[NSNotificationCenter defaultCenter] postNotificationName:[DatabaseNotificationName  LongLivedTransactionChanges] object:weakSelf.longLivedReadOnlyConnection userInfo:@{[DatabaseNotificationKey ConnectionChanges]:changes}];
               }
    }

We have observed the notification, its getting fired when Notification service extension modify any thing in Database, but its not giving any information related to changes it return an empty array. Any suggestions? any idea?

Aadil.

[Anupam824]

Hi @robbiehanson We ran into the same issue while implementing Notification extension. We have enabed the "enableMultiProcessSupport", all YapDatabaseView's are Non-persistent.

We added observer on "YapDatabaseModifiedExternallyNotification",

YapDatabaseCrossProcessNotification* cpn = [[YapDatabaseCrossProcessNotification alloc] initWithIdentifier:@"crossnotification"];
[self.database registerExtension:cpn withName:@"crossprocess"];

[[NSNotificationCenter defaultCenter] addObserverForName:YapDatabaseModifiedExternallyNotification object:self.database queue:[NSOperationQueue mainQueue] usingBlock:^(NSNotification * _Nonnull note) {
       NSArray <NSNotification *>*changes = [weakSelf.longLivedReadOnlyConnection beginLongLivedReadTransaction];
               if (changes != nil) {
                   [[NSNotificationCenter defaultCenter] postNotificationName:[DatabaseNotificationName  LongLivedTransactionChanges] object:weakSelf.longLivedReadOnlyConnection userInfo:@{[DatabaseNotificationKey ConnectionChanges]:changes}];
               }
    }

We have observed the notification, its getting fired when Notification service extension modify any thing in Database, but its not giving any information related to changes it return an empty array. Any suggestions? any idea?

Aadil.

@robbiehanson we are also facing the same issue. @maksadavid have you guys solved this issue finally?

[maksadavid]

We are not using YapDatabaseCrossProcessNotifications. In addition to the fixes that I mentioned above we are also executing a full reload of the mappings and the ui when the application becomes active. Usually the notification extension isn't modifying the database when the host app is active. I don't think that you can use non-persistent database views. In the documentation it is listed in the limitations. https://github.com/yapstudios/YapDatabase/wiki/Multiprocess-Support.

[Anupam824]

Thanks @maksadavid for your quick response. Can you please provide the code snipped of the fix which you have done.

[maksadavid]

I call this method when application becomes active. It should sync the mappings with the database and the ui.

- (void)moveMappingsToLastCommit {
    [self.databaseConnection beginLongLivedReadTransaction];
    [self.databaseConnection readWithBlock:^(YapDatabaseReadTransaction *transaction) {
        [self.mappings updateWithTransaction:transaction];
    }];
    [self.tableview reloadData];
}

In YapDatabaseViewTransaction.m. Line 229 I set BOOL shortcut = NO; I am using YapDatabase/SQLCipher 3.1.4 with Cocoapods.

[Anupam824]

I call this method when application becomes active. It should sync the mappings with the database and the ui.

- (void)moveMappingsToLastCommit {
    [self.databaseConnection beginLongLivedReadTransaction];
    [self.databaseConnection readWithBlock:^(YapDatabaseReadTransaction *transaction) {
        [self.mappings updateWithTransaction:transaction];
    }];
    [self.tableview reloadData];
}

Thanks @maksadavid for your quick response and your help is really appreciated. We will try this.

[Anupam824]

- (void)moveMappingsToLastCommit {
    [self.databaseConnection beginLongLivedReadTransaction];
    [self.databaseConnection readWithBlock:^(YapDatabaseReadTransaction *transaction) {
        [self.mappings updateWithTransaction:transaction];
    }];
    [self.tableview reloadData];
}

Thanks @maksadavid it worked for us. We have called from YapDatabaseModifiedExternallyNotification too which fixed it in all the cases. Thank you very much for your help.

[afriedmanGlacier]

We are having the same issue described, but haven't been able to make the solution work so far and have a few questions:

We are NOT writing to the database in the Notification Service Extension. Since we are doing just reads, we did not register all the same extensions in the extension that we have in the main app. We do have "enableMultiprocessSupport" enabled.

However, we did need to register a YapDatabaseRelationship extension to retrieve some information. Just registering the extension causes a write, and it crashes when we reopen the main app after the extension runs. If I remove this, no writes are performed and no problem.

1. If we use the solution by @maksadavid, do we also need to include ALL the same set of Yap extensions in the Notification Service Extension as are in the main app? We are trying to avoid that in order to keep the app extension as simple as possible. Including extensions means making more classes available to the extension.

2. The solution by @maksadavid does not make use of YapDatabaseCrossProcessNotifications. It seems that cross process notifications were created for this situation, but we haven't been able to figure out how to use them correctly. @robbiehanson is there a more complete description of this somewhere? The https://github.com/yapstudios/YapDatabase/wiki/Multiprocess-Support mentions it, but doesn't go into detail.

[afriedmanGlacier]

We did finally get YapDatabaseCrossProcessNotifications working and update the mappings when we receive that notification in the main app. This seems to work, so far we are no longer crashing immediately there.

Now, after a few (random number) of iterations of backgrounding the app, receiving push notification, etc, it crashes. This can happen whether we stay backgrounded primarily executing the Extension or if we open back up the main app and background again. We realize this could be completely unrelated to this issue.

I'd still be interested to know, since just registering an extension performs a write, are we still expected to include ALL the same set of Yap extensions in the Notification Service Extension as are in the main app?