YapDatabase icon indicating copy to clipboard operation
YapDatabase copied to clipboard

beginLongLivedReadTransaction and wal truncation doesn't work with multiprocess support

Open ksuther opened this issue 6 years ago • 0 comments

I'm investigating using YapDatabase's multiprocess support and it's working quite well except for a problem with long-lived read transactions and wal truncation.

In summary, here are the two problems I'm seeing:

  • wal checkpointing fails with SQLITE_BUSY if a long-lived read transaction is active. This results in the wal never getting truncated.

  • beginLongLivedReadTransaction doesn't return notifications since the changesets are occurring in a different process. This renders long-lived read transactions mostly useless if writes are performed in another process.

The wal problem appears to be because -[YapDatabaseConnection resetLongLivedReadTransaction] returns NO because snapshot == [database snapshot] is false. This causes the long-lived read transaction to not get moved forward to the latest snapshot, so the wal truncation fails.

I believe the snapshot isn't being updated because pendingAndCommittedChangesetsSince:until: is returning nil when encountering out of process changes. This eventually results in the connection's snapshot not getting moved forward (I think). If I stop all long-lived read transactions, manually trigger a checkpoint, then restart the long-lived read transactions then the wal truncation succeeds.

The issue with beginLongLivedReadTransaction feels similar. Since the changesets are being generated in another process the transaction isn't able to observe the changesets.

I may be off the beaten path in terms of the original implementation because YapDatabase is being used in multiple running processes on the Mac rather than on iOS where only one process is active at one time (usually). At the very least it'd be good to document these limitations. If I know this is expected behavior I can look in to sharing changesets between processes using XPC or something else. Assuming I'm not missing something obvious, I believe using XPC to synchronize the YapDatabaseConnection changesets would fix both of the problems I'm seeing here as well as allow preserving the YapDatabaseConnection caches instead of having to reset them every time a change is made in another process.

ksuther avatar Aug 14 '18 16:08 ksuther