NixOS
Support fine-grained database schema migrations
Motivation
Backward-compatible schema changes (e.g. those that add tables or nullable columns) now no longer need a change to the global schema file (/nix/var/nix/db/schema). Thus, old Nix versions can continue to access the database.
This is especially useful for schema changes required by experimental features. In particular, it replaces the ad-hoc handling of the schema changes for CA derivations (i.e. the file /nix/var/nix/db/ca-schema).
Schema versions 8 and 10 could have been handled by this mechanism in a backward-compatible way as well.
Context
Priorities and Process
Add :+1: to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.
Does this also help with the deadlock that are seeing when enabling ca derivations? Currently it requires a reboot of the machine because existing running nix processes will not allow the schema migration from happening as they hold the global lock file.
@Mic92 It should help with the deadlock since it doesn't hold the global lock anymore while doing SQLite schema migrations. (Maybe it should, but AFAIK SQLite doesn't need it.)
Great that's my main road block to deploy this by default on a larger amount of machines
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2024-10-16-nix-team-meeting-minutes-187/54835/1
To me, this looks good. Haven't tested it out though, also not sure how one might test this.
On a system without ca-derivations, enable ca-derivations and see if they work.
I don't yet fully understand the full lifecycle here.
For example:
-
What does one do if they need to make a "breaking" change to the CA schema while its experimental? It is non-breaking from the non-experimental status quo, but breaking from the last CA change.
-
What does one do if CA derivations is merged (not experimental) and then we make some sort of cross-cutting breaking change?
I think what this change is doing is grafting on a powerset of feature names in top of the last breaking migration, but I don't think that is sufficient to make this sustainable.
I don't yet fully understand the full lifecycle here.
For example:
1. What does one do if they need to make a "breaking" change to the CA schema while its experimental? It is non-breaking from the non-experimental status quo, but breaking from the last CA change. 2. What does one do if CA derivations is merged (not experimental) and then we make some sort of cross-cutting breaking change?I think what this change is doing is grafting on a powerset of feature names in top of the last breaking migration, but I don't think that is sufficient to make this sustainable.
We should not break the schema, experimental or not. We would append a migration after 20220326-ca-derivations if we need to update it. And when we remove the experimental feature status we will remove the if statement and apply this migration unconditionally. If we drop the feature, we would just ignore those extra columns that some nix installations might have in their database. Of course we need to make sure that we don't have different experimental features that introduce conflicting schemas, but we need to make sure this doesn't happen in any case.
@Mic92 Agreed. my understanding is this fine-grained approach assumes each named migration is independent of any other. It works for 2^n schemas freely generated by n non-conflicted migrations, but breaks down for when the partial order is not such a free semi-lattice (power set).
@Mic92 Agreed. my understanding is this fine-grained approach assumes each named migration is independent of any other. It works for
2^nschemas freely generated bynnon-conflicted migrations, but breaks down for when the partial order is not such a free semi-lattice (power set).
Well assuming we don't go crazy with experimental features, we should be able to test all combinations of migrations.
This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2024-11-20-nix-team-meeting-minutes-196/56688/1