CloudKit - Support for limiting change set size in a CKReference-safe manner

[gcox]

Resolves #212 Related: #268, #328

NOTE: This branch is based off the branch used in #510 so it could be tested

Changes

This change limits change set sizes such that all related records in a change set are kept together and that no change set can have more than a configurable max number of changes. The latter is copied from the changes in #268.

Examples

Example 1: Records being modified: [A, B, C, D, E, F, G, H, I, J, K] References: None Resulting change sets: [[A, B, C, D], [E, F, G, H], [I, J, K]]

Example 2: Records being modified: [A, B, C, D, E, F, G, H, I, J, K] References: D -> F Limit = 4 Resulting change sets: [[A, B, C], [D, E, F, G], [H, I, J, K]

Concerns/questions

If a parent and child record are modified, are they always sequentially close to each other? IOW, if the max change limit is 400, and 500 objects are imported, including records Foo and Bar, and Foo references Bar, is it possible that enumerating the dirty records will find record Foo at index 0 and record Bar at index 500? If so, this solution doesn't cover that scenario.

When processing dirty records, does the order of the records in the resulting change sets matter? If not, we could refactor this so that when generating the change sets shift referenced records so they are next to their parents to avoid the above edge case.

TODO

• [ ] Tweak the implementation so that we're not forcing child records to be in the same change set with their parent records unless those child records are new
• [ ] Add tests once a general solution is agreed on. Whatever we end up with is going to add too much complexity not to be tested.

[gcox]

@robbiehanson Your feedback on the 'Concerns/Questions' section would be appreciated. Then I can finish this up with some tests.

[robbiehanson]

There might be a bug here:

// Build a map of dirty record CKRecordIDs to all of their associated CKReferences'
NSMutableDictionary<CKRecordID *, NSMutableArray<CKRecordID *> *> *referenceMap = NSMutableDictionary.new;

[parentConnection->dirtyRecordTableInfoDict
   enumerateKeysAndObjectsUsingBlock:^(NSString * _Nonnull key,
                                       YDBCKDirtyRecordTableInfo * _Nonnull obj,
                                       BOOL * _Nonnull stop) {
if (obj.dirty_record.parent) {
    NSMutableArray<CKRecordID *> *parentReferences =
      (referenceMap[obj.dirty_record.parent.recordID] ?: NSMutableArray.new);

    [parentReferences addObject:obj.dirty_record.recordID];
    referenceMap[obj.dirty_record.recordID] = parentReferences;
  }
}];

For referenceMap is the key supposed to be obj.dirty_record.recordID or obj.dirty_record.parent.recordID?

[gcox]

@robbiehanson oh yes that’s a bug, should be obj.dirty_record.parent.recordID

[robbiehanson]

If a parent and child record are modified, are they always sequentially close to each other? IOW, if the max change limit is 400, and 500 objects are imported, including records Foo and Bar, and Foo references Bar, is it possible that enumerating the dirty records will find record Foo at index 0 and record Bar at index 500?

I think this is possible because we're currently storing changes in a dictionary (parentConnection->dirtyRecordTableInfoDict). And a NSDictionary doesn't give us any guarantees about order when we enumerate it.

When processing dirty records, does the order of the records in the resulting change sets matter?

Great question. I would say the order doesn't matter, but that it would be preferable to maintain the order when possible. For example:

Records being modified: [A, B, C, D, E, F, G, H, I, J, K] References: A -> K Limit = 4 Resulting change sets: [[A, K, B, C], [D, E, F, G], [H, I, J]

So it sounds like parentConnection->dirtyRecordTableInfoDict is not going to give us everything we need. We also want to store the order of the records. So either we convert the dictionary to an array, or store an array alongside the dictionary to record the order?

[gcox]

@robbiehanson Thanks for the input. I’ll take a look at both options.

[gcox]

@robbiehanson I believe the latest changes will produce this:

Records being modified: [A, B, C, D, E, F, G, H, I, J, K] References: A -> K Limit = 4 Resulting change sets: [[A, K, B, C], [D, E, F, G], [H, I, J]

Gotta figure out a reasonable way to test this area though