elasticsearch
elasticsearch copied to clipboard
elastic
Calculate minimum compat and ccs versions from the release version mapping
Use the release version mapping to calculate the minimum transport and ccs versions, rather than hard-coding them
This is running into problems because 8.13.0 has just been frozen, but not finalized - so there is a Version, but no defined TransportVersion. The options are:
- Check the previous previous Version if there is no mapping for the previous Version
- Add a provisional transport version mapping on release freeze, rather than waiting for finalize to add the mapping
1 is easier, but could give the incorrect version if something goes wrong with the mapping. 2 requires complicated changes to the build infrastructure around mappings
Another option is that instead of calculating it at runtime, we do it at compile time using the release automation (which could be combined with updating the index/transport version on minor releases). I'll look at this instead.
1 is easier, but could give the incorrect version if something goes wrong with the mapping.
Will this be transient? I.e. when 8.13.0 is finalize, will we re-generate/overwrite the mapping and have the correct version again?
Yes, it'll be correct again after finalize, which is a few days after freeze. This should only be the case on main, but there's no way to check that from within the code itself.
Started work on build automation solution in https://github.com/elastic/elasticsearch/pull/105550
- Add a provisional transport version mapping on release freeze, rather than waiting for finalize to add the mapping
Why would it be provisional? We are freezing, so we shouldn't expect it to change, we would only change it if a new transport or index version had to be added to the release branch due to a bug. But under normal circumstances, it should stay the same through finalization.
2 requires complicated changes to the build infrastructure around mappings
What complications exist?
Also, I've found it a little difficult to follow when and why ccs version is updated. It would be helpful to have a concrete example, with a timeline, so that we can all have a shared mental model.
The published version numbers are set on finalize, precisely because there might be bug fixes needed between freeze and finalize. If we ever did need to introduce a fix version, and the provisional version was already recorded, we would need to remember to update the version in the mapping file too - which we would likely forget to do, leading to incorrect mappings (unless the automation handled updating it).
Setting a provisional version and then modifying it with automation requires changes to the mapping tasks and automation to modify existing mappings, not just add new ones. Currently the code only adds new mappings, modifying existing ones would be a different operation requiring new tasks and/or options.
I prefer https://github.com/elastic/elasticsearch/pull/105550 anyway, over this, as it is a lot simpler, and doesnt have all these problems.
Just my 2c: I also prefer https://github.com/elastic/elasticsearch/pull/105550. Not only is a bit simpler, it seems the more appropriate place - we know the min CCS transport version on release, and it's a constant that is not going to change.
there might be bug fixes needed between freeze and finalize
If there is a bugfix change to transport versions, wouldn't we need to update the min ccs at the same time so that tests don't start failing?
we would need to remember to update the version in the mapping file too - which we would likely forget to do
The mapping file must be up to date, if we get that wrong, our error messages are wrong. If there's a way for them to be out of sync, we should have a test for that, to ensure it can't happen.
I prefer https://github.com/elastic/elasticsearch/pull/105550 anyway, over this, as it is a lot simpler, and doesnt have all these problems.
https://github.com/elastic/elasticsearch/pull/105550 to me seems to explicitly allow things to be out of sync, which I don't think we should ever allow.