neos
10.0 FEATURE: Compact migrations
As discussed with @kitsunet , @kdambekalns and @dlubitz
Current state: We have a lot of database migrations collected over the last years, which contain already outdated migrations, but get executed on each new installation of Neos. This leads to issues with database server compatibility like mentioned above.
We want to rollup all migrations into one single migration per package in next major version Neos/Flow 10.0 and remove all old migrations.
To achieve this, we need to ensure that all database instances have a defined state.
Our idea is to add a blank migration, as latest as possible, in Neos 9.3. So we can add a migration for a clean database setup, with all table creations into 10.0 with the same version name. This would skip this migrations in Neos 10, if it was already executed in 9.3 (upgrade). If the version was not yet executed, we assume no migration has been run, so we can run the 10.0 migrations with creation of all tables (new install).
External packages might need to release a new major version, to provide dedicated migrations if needed.
Needed documentation:
- Before upgrading to Neos 10 you MUST apply all migrations in Neos 9.3.
- In Neos 10.0 you might get a warning, that applied migrations are not available anymore -> You need to run the cleanup of doctrine migrations.
For testing purpose I created some draft PRs to show what we need to do and document some learings.
Neos.Flow: https://github.com/neos/flow-development-collection/pull/3506 Neos.Party: https://github.com/neos/party/pull/36 Neos.Media & Neos.Neos: https://github.com/neos/neos-development-collection/pull/5655
Following learnings:
Postgres
Remember! There are also postgres migrations.
Empty Migration in previous Package Version
We need to put a empty migration into the package version BEFORE the package version with the compacted migrations. This ensures that we don't re-apply migrations to an existing database schema. If this empty version was already applied in the latest version, nothing has changed, but we can use this execution marker to "skip" the compacted migration in the next major version.
Major release necessary
With a new major release of each repository, we can also define the dependencies with version constraints. This is necessary to ensure the right order in the execution of migrations (see below).
We have a clear "breaking" point, which ususally has a more complex upgrade path anyways, compared to minor releases. It's also easier to communicate, that you need to upgrade until last minor/bugfix version of the latest version, before upgrading to next major.
!!! Package dependencies on database level
The packages have dependencies between each other. This is also reflected on the database. Some migrations need others to got applied earlier to create foreign key constraints.
- Neos.Neos requires Neos.Media
- Neos.Neos requires Neos.Party
- Neos.Party requires Neos.Flow
This need to get reflected in the version numbers of each package, to ensure the execution in the right order. Moreover we should think about possible issues due to not or later installed packages.
Manual migrations independent of doctrine enties
There are also database migrations added, which are not "generated" by doctrine based on given entities. With Neos 9.x we introduced non-orm entities, but have used the doctrine migrations to ensure the database schema.
- neos_neos_workspace_metadata
- neos_asset_usage
- neos_neos_workspace_metadata (changes)
- neos_neos_impending_hard_removal_conflict
- ... maybe more new ones