Automatic handling of backlinks when migrating to EmbeddedObject

[DominicFrei]

Summary

Migrating a table from non-embedded to embedded is not really easy and obvious in the user-facing SDKs. This could be improved by automating the main problems and making the migration easier for the user.

Motivation

When migrating a table from non-embedded to embedded users do struggle to get it right because they have to deal with two cases:

• orphaned objects (no backlinks at all) - https://github.com/realm/realm-core/blob/master/src/realm/table.cpp#L1060
• too many backlinks - https://github.com/realm/realm-core/blob/master/src/realm/table.cpp#L1063

The struggle itself has two reasons:

• It's not really all to obvious how to actually handle that during the MigrationBlock (there are multiple questions about how to do that in Cocoa and .NET resulting in https://github.com/realm/realm-dotnet/issues/2408 and https://github.com/realm/realm-cocoa/issues/7145).
• In some cases it simply is not possible at the moment (in .NET we disallow querying EmbeddedObject in general which means they can't even be iterated while in a migration).

While allowing the latter for the special case of migration would partial solve this, I'd like to propose the following changes to solve all those problems at once.

Goals

When applying the post migration changes we ensure the following during https://github.com/realm/realm-core/blob/master/src/realm/object-store/object_store.cpp#L832:

• Objects with too many backlinks get duplicated and assigned one to each backlink so that we automatically end up with objects with only one backlink.
• Orphaned objects will be deleted. This will only happen when a flag is explicitly set to make sure users consciously confirm that data should be deleted.

User-facing API

• Adding a new option deleteOrphanedObjects to the migration, type bool. In the SDKs this would probably make most sense in RealmConfiguration since the MigrationBlock and the DeleteFilesOnMigration flag already live there. In Core this might probably be best situated in ObjectSchema next to IsEmbedded itself.
• Default would be false so that it does not influence any existing migrations.

[niralishaha25]

Any update when this would be released? Unable to use Embedded objects because of the error - "At least one object does not have a backlink (data would get lost)."

[DominicFrei]

Hi @niralishaha25,

this is just an improvement to automate things that already work manually. So, fortunately this feature is not necessary to solve your problem.

Could you please create a ticket in the SDK you're using describing the problem in more detail and show some sample code? We can then help you solve this problem.

Thanks!

[niralishaha25]

I couldn't find any API related to migrating embedded objects though went through multiple issues on GitHub. Thought this is the one which actual has user facing API. Continuously facing error: "At least one object does not have a backlink (data would get lost)" while accessing realm

[DominicFrei]

@nirinchev Which SDK are you using? If we know which language you are looking for we probably point you into the right direction. I've also created a little Q&A document for EmbeddedObject since this type seems to produce questions regularly: https://github.com/realm/realm-dotnet/discussions/2594

[niralishaha25]

I am using realm-cocoa for iOS version 10.13.0

[DominicFrei]

@niralishaha25 The Cocoa repository does have examples for migrations. You can also look through the list of issues related to your exception. And on top of that feel free to create a new issue if this doesn't solve the problem. 👍

[sipersso]

@DominicFrei The sample doesn't show anything about how to do this migration. @niralishaha25 If you look at https://github.com/realm/realm-cocoa/issues/7145 there is some info there that could be useful. Migrating to embedded objects is far from trivial I am afraid :/

[sipersso]

After investigating this further, it looks like the case of removing multiple backlinks by copying data and assigning exactly one backlink isn't possible in the iOS SDK for the moment.

I didn't understand the API to create embedded objects during migration so added a feature request here: https://github.com/realm/realm-cocoa/issues/7508. But apparently it was re-labeled as a bug, so I don't think this is possible to do. In the Android SDK I have been able to do this without problems.

[DominicFrei]

Hi @sipersso,

the other issue you created was turned into a bug because it is already supposed to work but doesn't. Hence bug not enhancement. It's just for our internal prioritization though, for you it's all the same I suppose. Something that you need is missing, sorry about that.

We'll take care of this in the other issue so you can go ahead a finish your migration including EmbeddedObjects with multiple backlinks. 👍

I'll keep this issue here open as well since it's the automatic handling of all of this which will probably happen sometime after fixing the bug. 👍