Schema versioning and migrations?

[lewisl9029]

I'm trying to figure out how to handle data schema versioning and migrations with remotestorage.

Is this something that remotestorage has implemented at the moment?

If so, where can I find some documentation or examples on how to create/perform migrations?

If not, is this somewhere on the roadmap? Or is it something developers are expected to implement on top of remotestorage on their own?

Thanks!

[raucao]

As of yet, this is an unsolved problem. The idea for a generic migration library/tool came up before, of course. I think ideally it should be an add-on for remoteStorage.js, which app developers can use within their apps somehow.

Afaik nobody currently plans to work on this, but a few people would likely be more than happy to assist someone who would start throwing around ideas and/or working on it.

[lewisl9029]

Thanks for the follow up.

The main concern I have about implementing this on my own is how best to handle synchronization between clients of conflicting versions. I think it might be best to simply disable synchronization when versions mismatch, but I'm unsure how to best implement this behavior on top of the library (maybe a separate module to store just version information and sync'ing and checking that before any other module?).

I'd love to hear from people with more working experience with remoteStorage or other data synchronization platforms on how to approach versioning and migrations. I feel like this is a fairly crucial piece of functionality to get right before launching anything into production.

[michielbdejong]

This question is about module development, right? OK, I came up with a system:

So say you have the pictures module and want to start allowing pictures that appear in two albums. The approach for this is to allow JSON-LD documents to appear in albums, and they contain a reference to the actual photo. This would indeed break clients that use the current module, where the assumption is safe that any file that is in an album, is an image file.

I would suggest the following procedure:

• first of all, nudge all users of the pictures module to update their apps. we are still only a handful of people, and this will still avoid the bulk of the potential user pain.
• the first time a user adds a photo to >1 album, display a warning saying "Need to upgrade your pictures to version 2.0, some older apps may display your pictures in the first album, not in the others"
• store a special file in the root of the module's data folder, which indicates "this account uses features from version 2.x of the pictures module"

In general: we can standardize that "highest data module version used" indicator now already, without knowning when and why we might at some point make breaking changes to any of the modules. We can now already make each module do a check for e.g. /.highestMajorVersion, and if it's anything higher than 1, display a warning to the user saying "This app may not display all your data correctly"

[raucao]

I think the idea of keeping a meta data file for this makes sense. And why not have one with a more general filename with all the information in it from the start?

[Ragnis]

Wouldn't it be cleaner to use a versioned JSON-LD ID instead of keeping the version info in a separate file?

On Tue, Apr 14, 2015 at 5:10 PM, Sebastian Kippe [email protected] wrote:

I think the idea of keeping a meta data file for this makes sense. And why not have one with a more general filename with all the information in it from the start?

— Reply to this email directly or view it on GitHub https://github.com/remotestorage/remotestorage.js/issues/858#issuecomment-92866700 .

[raucao]

Yes, but that only helps you with JSON objects and doesn't solve the problem of changing directory structures for example.

Edit: Just to be clear, I still think that's a very good idea for JSON objects/schemas and other files don't need to be versioned anyway. Right?

[raucao]

Turns out we (at @5apps) need to migrate some data in commercial RS-enabled apps of one of our clients soon, so we need to solve this on the way. Timeframe is about 1-2 months from now.