Separate "upgrade/conversion" migrations from "install" migrations

[alexweissman]

It has become clear that our approach for new installations of a given project (including a bare install of UF without any additional Sprinkles), which consist of a series of migrations starting from the very first version, is insufficient.

There is not always a clean (lossless) way to transform data in a database from an older schema to a newer one. For example, going from UF3 to UF4, we got rid of "group" and "primary group", replacing them with "role" and "group". Our current upgrade tool does the best job that it can, but the fact of the matter is that it wouldn't make sense for new UF4 installations to start with a UF3 schema and then migrate that to the UF4 schema. It makes far more sense to simply set up the UF4 schema directly in the case of new installations, and only perform a migration/upgrade when necessary with legacy projects.

For this reason, we need to separate the concepts of "installing" and "upgrading" a database schema. I would argue that an "upgrade" is a far more complex process, which may still end up invoking the installation code of the target version, but will have many other things to do as well.

I'd say an "upgrade" should be able to declare a source version, and target version, of the project's schema. An "installation" need only declare a target version.

There are some other questions to answer before we can make this happen, like:

1. Can upgrades be chained? For example, can you invoke a UF2->UF3 upgrade, and then a UF3->UF4 upgrade, or would it be better to just have a direct UF2->UF4 upgrade? This could end up exploding combinatorily as we release more versions of UF.
2. How do we divide upgrades across many Sprinkles? Upgrades may involve some tight dependencies among tables originally created by different Sprinkles.

[amosfolz]

Were there some changes made in 4.2 that would impact this?

[lcharette]

Were there some changes made in 4.2 that would impact this?

Not that I know of. Seeds where added, which probably helped in some cases.

I think the real solution to @alexweissman original issue is probably import. In a future where UFx db schema is not compatible with UF4 one, a import bakery command to copy data from a base 4.x.x install to a new fresh 5.x.x install might be best.

This issue was focused on the UF3 -> UF4 upgrade (it was a big one). At this point, I'm not sure we'll need such a big update so soon, on the db side of things at least, and for 5.0.0 too.